This patch fixes the m4 port by removing unneeded make subdirs that would cause the build to fail on some systems. We now only care about the `lib` and `src`.
