Basically what Mary just said ...
But I want to elaborate a little. It would help to know what version of Moodle you are using, and how long the videos are, to answer your question.
In Moodle 2.3 the assignment changed and so did some of your options. But simply put, you can offer your students the option to upload a file, as the assignment submission.
If you are uploading videos (ie not shooting them from a webcam) then MP4 is probably the best format. That is because it plays back on most devices and is well supported by online video players. But asking your students to convert their own files might be tricky. So it will probably depend on how you are asking them to record their videos.
File size depends on your server, the distance from your students to the server, and the length of the videos, and how they are recording them. But in general small is good. When shooting from a webcam, I aim to keep file sizes no greater than 1.5M per minute.
Actually I worked with an organisation earlier this year who needed very long video files uploaded as assignment submissions. We made a Youtube assignment submission type. The plan is to release that back to the community. This would clear up the need to worry about file formats or file size.