The entrance to a cave significant enough to get a number and a survey will eventually be marked by a numbered tag attached to a spit. This will then become the primary survey station - ie. the point where an underground survey will start, and the point to which a surface survey should go. It's worth thinking about where you would put such a tag right from the start. Unlike the first rigging bolt (often used as the first point of a survey in the past) it should be sited with a particular view to its visibility and accessibility without having to put on SRT kit. If such a point has a clear view of the majority of the sky, then this is the point to use for a GPS fix too.
If however, you are dealing with a cave at the foot of a cliff, or otherwise with a restricted view of the sky, then choose instead a good landmark with a wide, clear view of the sky, and within one (or maybe two) survey shots of the entrance. If you have found a group of caves close together, it might be better to GPS a central point rather than get quick (but less accurate) fixes on each entrance.
We now (2018) have differential GPS which is much more accurate than in the past (e.g. Wookey's 1996 article) but altitudes are often very inaccurate and GPOS devices don't tell you how inaccurate the altitude is.
Once you have chosen your point, mark it in some way (could be a spit hole or a cairn, for example - we aren't supposed to use paint any more) and place the GPS on the point. If you build a cairn, make it wide rather than high - tall cairns are knocked down by the depth of snow each winter. Give it a couple of minutes to get a fairly good fix (the first figure reported may be quite a way out, but after a couple of minutes things should settle). Then mark the point as a waypoint. If you're feeling really keen, you can set it up for averaging, which gives a more accurate fix – some GPS receivers support this automatically, and with others you can just leave it recording a track log, then record another waypoint at the same place just before you leave so it's clear to someone examining the track log when you actually left. (Averaging was once crucial to getting any kind of remotely close fix, but is somewhat less important these days now that Selective Availability has been turned off.)
While the GPS is recording your location, you can do something useful (like rigging the cave, doing a surface survey from the GPS point to the marker spit, looking for other caves, or even having lunch!) Remember to stop the waypoint averaging before moving the unit or changing the display page. Take a photo of your GPS point showing at least one of your cave entrances too.
We all use the same coordinate system WGS84 these days, so the extensive discussion on coordinate systems has been moved to a different page. If you are really interested you can read Olaf's articletoo.