The OS you wish to work on will be the main decider of what tools you use to develop with, if you are aiming at Linux then NASM, JWASM or GAS using Intel notation are probably the best choices. If you are targetting a Windows platform, you have the choices of MASM and compatibles (JWASM), FASM and NASM. TASM is just too old. Of this choice the best support and range of code examples is in MASM and compatible assemblers, FASM is an excellent assembler and very powerful once you know how to use it but it does not have the support base of MASM. NASM is also a well written powerful assembler but it is more pointed at cross platform compatibility that ease of use.
As far as instructions to learn, always start with the basic components, MOV, ADD, SUB, CMP, TEST, ROL/R, etc .... as these common instructions do the lion share of the work. SIMD instructions are specialised and targetted at streaming multimedia data and while they are often used for other purposes, you really need to know the basic stuff first. Get used to the complex addressing modes "mov eax, [ecx+edx*4+64]" style notation and the instruction component order "mov eax, 1234" means move the immediate value of 1234 into the EAX register.
It is a lot to learn but its a ton of fun once you get the swing of it and there is real pace and power in mastering this style of code.