Scholarly books use technical language, are written by experts for experts, and employ "scholarly apparatus" such as detailed references giving the sources of information used.
There are also books which, although they are not "scholarly" in the same sense, are highly specialized, and are intended for a reader in a specific field. (An example is Safe Operation of Fire Tankers, published by the U.S. Fire Administration.)
There is no "checkoff" option in the library search to determine scholarly/popular status for books. In general, you must look at the book, examine its language, and determine its intended audience in order to determine whether it is popular or scholarly/field specific.
Need help determining if a book is popular or scholarly? Ask a librarian.