1. A piece of hardware that encapsulates some function but can't be used without some kind of front end. Today we have, especially, `print engine': the guts of a laser printer. 2. An analogous piece of software; notionally, one that does a lot of noisy crunching, such as a `database engine'.
The hacker senses of `engine' are actually close to its original, pre-Industrial-Revolution sense of a skill, clever device, or instrument (the word is cognate to `ingenuity'). This sense had not been completely eclipsed by the modern connotation of power-transducing machinery in Charles Babbage's time, which explains why he named the stored-program computer that he designed in 1844 the `Analytical Engine'.