Richmond and Washington are about 97 miles apart by road. Split the difference and you arrive near Philadelphia, the city closest to the halfway point between them. That puts roughly 4 hr 20 min of driving on the Richmond side and 3 hr 5 min on the Washington side — the fairest single meeting point among the cities near the middle.
At this range, Philadelphia works for a same-day meetup: close enough from both Richmond and Washington to meet for lunch or an afternoon and still be home by evening.
If Philadelphia doesn't have what you're after, Raleigh and Pittsburgh are also close to the midpoint and worth a look — each keeps the drive reasonably balanced between Richmond and Washington.
City descriptions adapted from Wikipedia (CC BY-SA); photos via Wikimedia Commons, credited above.