New York and Richmond are about 286 miles apart by road. Split the difference and you arrive near Philadelphia, the city closest to the halfway point between them. That puts roughly 2 hr 1 min of driving on the New York side and 4 hr 20 min on the Richmond side — the fairest single meeting point among the cities near the middle.
At this range, Philadelphia works for a same-day meetup: close enough from both New York and Richmond to meet for lunch or an afternoon and still be home by evening.
If Philadelphia doesn't have what you're after, Washington and Hartford are also close to the midpoint and worth a look — each keeps the drive reasonably balanced between New York and Richmond.
City descriptions adapted from Wikipedia (CC BY-SA); photos via Wikimedia Commons, credited above.