How to grep a log file for words listed in separate text file?

Hello,

I want to grep a log ("server.log") for words in a separate file ("white-list.txt") and generate a separate log file containing each line that uses a word from the "white-list.txt" file.

Putting that in bullet points:

  • Search through "server.log" for lines that contain any word contained within "white-list.txt" file
  • Create separate file ("player_activity.log")

I've done this before with this command line:

grep -E "player1|player2|player3" server.log >player_activity.log

But I'm tired of modifying that line and would just like to draw from my known player list in a different file. I want it to substitute the "player1|player2|player3" with looking through a white-list.txt that looks like:

player1
player2
player3
player4
player5
player6

The output should look like this:

2013-08-11 08:46:23 [INFO] player1[/xxx.xxx.xxx.xxx:53531] logged in with entity id 1072 at ([regolare] -220.7308956951062, 62.0, -108.79669524885138)
2013-08-11 08:46:23 [INFO] Player joined: player1
2013-08-11 08:50:50 [INFO] player2[/xxx.xxx.xxx.xxx:53019] logged in with entity id 5685 at ([mundusprime] 1995.377727365253, 75.0, 883.1106329044326)
2013-08-11 08:50:50 [INFO] Player joined: player2
2013-08-11 08:51:00 [INFO] <player1> morning
2013-08-11 08:51:06 [INFO] <player2> hola mi compadre!
2013-08-11 08:51:31 [INFO] <player2> how goes the pyramid?
2013-08-11 08:51:53 [INFO] <player1> roof is complete, need to finish clearing interior.
2013-08-11 08:52:06 [INFO] <player2> I took a peek the other day, it's impressive!
grep -f white-list.txt server.log

white-list.txt:

player1
player2
player3
player4
player5
player6
1 Like

OK... That was way too simple. Thanks.

OK... Now I have a new spin-off issue. Is there a way to grep "file a" (a log file) for names held in "file b" (a list of usernames) AND also file "c" (a list of specific phrases of interest that could show up in the log - i.e. showing login attempts from non approved users)?

I would do it as two separate grep statements, but I would like the results to be mixed together in chronological order from the log file.

$ grep -f names <(grep -f phrases mainfile)

So, "names" would be my "white-list.txt" full of usernames?
And "phrases" would be my list of phrases of concern?
And "mainfile" would be the server.log that I'm parsing?

That's the theory :slight_smile:

I did this:

grep -f white-list.txt <(grep -f activity-list.txt server.log) > temp2.log

It didn't find anything within the "activity-list.txt" file.

Show a sample of each of the files where you expected a match.

Matches are only made when the user appears in the white list and the phrase is present in the activity list. If you want to match either the user or the activity, revert to the previous, simpler grep and put everything - the usernames and the phrases - in the white list file.

Bear in mind, too, that grep is case-sensitive without the -i option. Some of the previous log file you posted seems to be mixed case.

Sample of server.log

2013-09-17 08:43:09 [INFO] Starting Minecraft server on *:25565
2013-09-17 08:43:09 [INFO] This server is running CraftBukkit version git-Bukkit-1.5.2-R1.0-b2788jnks (MC: 1.5.2) (Implementing API version 1.5.2-R1.0)
2013-09-17 08:43:10 [INFO] [PermissionsEx] sql backend registered!
2013-09-17 08:43:10 [INFO] [PermissionsEx] file backend registered!
2013-09-17 08:43:10 [INFO] [PermissionsEx] PermissionEx plugin initialized.
2013-09-17 08:43:10 [INFO] [PhysicalShop] Loading PhysicalShop v9.2.3
2013-09-17 08:43:10 [INFO] [LagMeter] Loading LagMeter v1.12.0
2013-09-17 08:43:10 [INFO] [AntiBed] Loading AntiBed v1.0.1
2013-09-17 08:43:10 [INFO] [SwearingBlocker] Loading SwearingBlocker v1.57
2013-09-17 08:43:10 [INFO] [SimpleAFK] Loading SimpleAFK v0.9.1
2013-09-17 08:43:10 [INFO] [Spawn Bed] Loading Spawn Bed v0.1
2013-09-17 08:43:10 [INFO] [SimpleSeen] Loading SimpleSeen v0.3
2013-09-17 08:43:10 [INFO] [WorldEdit] Loading WorldEdit v5.4.5
2013-09-17 08:43:10 [INFO] [TimeFold] Loading TimeFold v1.0
2013-09-17 08:43:10 [INFO] [Vanilla] Loading Vanilla v0.2
2013-09-17 08:43:10 [INFO] [RewardMe] Loading RewardMe v2.5.1
2013-09-17 08:43:10 [INFO] [WeatherRestrictions] Loading WeatherRestrictions v2.6.3
2013-09-17 08:43:10 [INFO] [Landmarks] Loading Landmarks v0.7
2013-09-17 08:43:10 [INFO] [Vault] Loading Vault v1.2.24-b304
2013-09-17 08:43:10 [INFO] [Multiverse-Core] Loading Multiverse-Core v2.5-b665
2013-09-17 08:43:10 [INFO] [Diving-Helmet] Loading Diving-Helmet v1.8.1
2013-09-17 08:43:10 [INFO] [PermissionsEx] Loading PermissionsEx v1.19.5
2013-09-17 08:43:10 [INFO] [PermissionsEx] Initializing file backend
2013-09-17 08:43:10 [INFO] Permissions file successfully reloaded
2013-09-17 08:43:10 [INFO] [Wither] Loading Wither v2.0
2013-09-17 08:43:10 [INFO] [MobCatcher] Loading MobCatcher v3.7
2013-09-17 08:43:10 [INFO] [Multiverse-Inventories] Loading Multiverse-Inventories v2.5-b346
2013-09-17 08:43:10 [INFO] [WormholeXTreme] Loading WormholeXTreme v1.031
2013-09-17 08:43:10 [INFO] [WormholeXTreme][v1.031] Loading WormholeXTreme ...
2013-09-17 08:43:10 [INFO] [WormholeXTreme][v1.031] Load complete
2013-09-17 08:43:10 [INFO] [Multiverse-NetherPortals] Loading Multiverse-NetherPortals v2.5-b660
2013-09-17 08:43:10 [INFO] [Vault] Enabling Vault v1.2.24-b304
2013-09-17 08:43:10 [INFO] [Vault][Permission] PermissionsEx found: Waiting
2013-09-17 08:43:10 [INFO] [Vault][Permission] SuperPermissions loaded as backup permission system.
2013-09-17 08:43:10 [INFO] [Vault][Chat] PermissionsEx found: Waiting
2013-09-17 08:43:10 [INFO] [Vault] Enabled Version 1.2.24-b304
2013-09-17 08:43:10 [INFO] [Vault][Permission] PermissionsEx hooked.
2013-09-17 08:43:10 [INFO] Preparing level "mundusprime"
2013-09-17 08:43:10 [WARNING] ----- Bukkit Auto Updater -----
2013-09-17 08:43:10 [WARNING] Your version of CraftBukkit is out of date. Version 1.6.2-R1.0 (build #2879) was released on Wed Sep 11 02:02:22 PDT 2013.
2013-09-17 08:43:10 [WARNING] Details: /downloads/craftbukkit/view/02342_1.6.2-R1.0/
2013-09-17 08:43:10 [WARNING] Download: /downloads/craftbukkit/get/02342_1.6.2-R1.0/craftbukkit.jar
2013-09-17 08:43:10 [WARNING] ----- ------------------- -----
2013-09-17 08:43:10 [INFO] Preparing start region for level 0 (Seed: 6416758407589632547)
2013-09-17 08:43:11 [INFO] Preparing start region for level 1 (Seed: -4634992794148121490)
2013-09-17 08:43:11 [INFO] Preparing spawn area: 83%
2013-09-17 08:43:11 [INFO] [PhysicalShop] Enabling PhysicalShop v9.2.3
2013-09-17 08:43:11 [INFO] [PhysicalShop] Did not hook into LogBlock
2013-09-17 08:43:11 [INFO] [PhysicalShop] PhysicalShop v9.2.3 enabled.
2013-09-17 08:43:11 [INFO] [LagMeter] Enabling LagMeter v1.12.0
2013-09-17 08:43:11 [INFO] [LagMeter] Not using logs folder.
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] Enabled! Polling every 40 server ticks. Logging to /home/mc/servers/bukkitpvt/plugins/LagMeter/lag.log.
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] Chunks loaded:
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] World "mundusprime": 625.
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] World "mundusprime_nether": 625.
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] Total chunks loaded: 1250
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] Entities:
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] World "mundusprime": 183.
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] World "mundusprime_nether": 89.
2013-09-17 08:43:11 [INFO] [LagMeter 1.12.0] Total entities: 272
2013-09-17 08:43:11 [INFO] [AntiBed] Enabling AntiBed v1.0.1
2013-09-17 08:43:11 [INFO] AntiBed enabled.
2013-09-17 08:43:11 [INFO] [SwearingBlocker] Enabling SwearingBlocker v1.57
2013-09-17 08:43:11 [INFO] [SwearingBlocker] Vault not found, turning economy features off
2013-09-17 08:43:11 [INFO] [SimpleAFK] Enabling SimpleAFK v0.9.1
2013-09-17 08:43:11 [INFO] [SimpleAFK] Loading Config files...
2013-09-17 08:43:11 [INFO] [SimpleAFK] Config file 'config.yml' loaded
2013-09-17 08:43:11 [INFO] [SimpleAFK] Config file 'language_en.yml' loaded
2013-09-17 08:43:11 [INFO] [SimpleAFK] ... Config files loaded
2013-09-17 08:43:11 [INFO] [SimpleAFK] SimpleAFK v0.9.1 Enabled
2013-09-17 08:43:11 [INFO] [Spawn Bed] Enabling Spawn Bed v0.1
2013-09-17 08:43:11 [INFO] [Spawn Bed] Spawn Bed Enabled
2013-09-17 08:43:11 [INFO] [SimpleSeen] Enabling SimpleSeen v0.3
2013-09-17 08:43:11 [INFO] [SimpleSeen] 0.3 enabled.
2013-09-17 08:43:11 [INFO] [WorldEdit] Enabling WorldEdit v5.4.5
2013-09-17 08:43:11 [INFO] WEPIF: Using the Bukkit Permissions API.
2013-09-17 08:43:11 [INFO] [TimeFold] Enabling TimeFold v1.0
2013-09-17 08:43:11 [INFO] [Vanilla] Enabling Vanilla v0.2
2013-09-17 08:43:11 [INFO] [RewardMe] Enabling RewardMe v2.5.1
2013-09-17 08:43:11 [INFO] [RewardMe] enabled
2013-09-17 08:43:11 [INFO] [WeatherRestrictions] Enabling WeatherRestrictions v2.6.3
2013-09-17 08:43:11 [INFO] WeatherRestrictions version 2.6.3 is enabled!
2013-09-17 08:43:11 [INFO] [Landmarks] Enabling Landmarks v0.7
2013-09-17 08:43:11 [INFO] [Landmarks] Loading /home/mc/servers/bukkitpvt/plugins/Landmarks/markers.json
2013-09-17 08:43:11 [INFO] [Landmarks] Successfully loaded 0 markers.
2013-09-17 08:43:11 [INFO] [Landmarks] Landmarks 0.7 enabled.
2013-09-17 08:43:11 [INFO] [Multiverse-Core] Enabling Multiverse-Core v2.5-b665
2013-09-17 08:43:11 [INFO] [Multiverse-Core] [AllPay] - Version 13.0 - using only an item based economy for Multiverse-Core v2.5-b665
2013-09-17 08:43:12 [INFO] Preparing start region for level 2 (Seed: 6416097921303314683)
2013-09-17 08:43:12 [INFO] Preparing start region for level 3 (Seed: 3044967559793989709)
2013-09-17 08:43:13 [INFO] Preparing start region for level 4 (Seed: 1910975946905473061)
2013-09-17 08:43:13 [INFO] Preparing start region for level 5 (Seed: 5361911335612553857)
2013-09-17 08:43:13 [INFO] [Multiverse-Core] 6 - World(s) loaded.
2013-09-17 08:43:14 [INFO] [Multiverse-Core] Version 2.5-b665 (API v18) Enabled - By Rigby, fernferret, lithium3141 and main--
2013-09-17 08:43:14 [INFO] [Diving-Helmet] Enabling Diving-Helmet v1.8.1
2013-09-17 08:43:14 [INFO] [Diving-Helmet] is Enabled
2013-09-17 08:43:14 [INFO] [PermissionsEx] Enabling PermissionsEx v1.19.5
2013-09-17 08:43:14 [INFO] [PermissionsEx] Superperms support enabled.
2013-09-17 08:43:14 [INFO] [PermissionsEx] v1.19.5 enabled
2013-09-17 08:43:14 [INFO] WEPIF: PermissionsEx detected! Using PermissionsEx for permissions.
2013-09-17 08:43:14 [INFO] [Vault][Chat] PermissionsEx_Chat hooked.
2013-09-17 08:43:14 [INFO] [Wither] Enabling Wither v2.0
2013-09-17 08:43:14 [INFO] [Wither] Wither v2.0 enabled!
2013-09-17 08:43:14 [INFO] [MobCatcher] Enabling MobCatcher v3.7
2013-09-17 08:43:14 [INFO] [MobCatcher] Author: Malikk
2013-09-17 08:43:14 [INFO] e[0;31;1mYou are not running the correct version of Craftbukkit for this version of MobCatcher.e[m
2013-09-17 08:43:14 [INFO] e[0;31;1mMobCatcher v3.7 is intended to be run on CB 1.5.1-R0.1-SNAPSHOTe[m
2013-09-17 08:43:14 [INFO] e[0;31;1mExpect major bugs or failure to enable. Please do not report bugs until you are using compatible versions of CB and MobCatcher.e[m
2013-09-17 08:43:14 [INFO] [MobCatcher] Hooked Vault 1.2.24-b304
2013-09-17 08:43:14 [INFO] [MobCatcher] Spout was not found.
2013-09-17 08:43:14 [INFO] [MobCatcher] remove stackability: true
2013-09-17 08:43:14 [SEVERE] Error occurred while enabling MobCatcher v3.7 (Is it up to date?)
java.lang.NoClassDefFoundError: net/minecraft/server/v1_5_R2/Item
    at me.jordan.mobcatcher.MobCatcher.removeStackability(MobCatcher.java:1170)
    at me.jordan.mobcatcher.MobCatcher.onEnable(MobCatcher.java:120)
    at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:217)
    at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:457)
    at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:381)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.loadPlugin(CraftServer.java:282)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.enablePlugins(CraftServer.java:264)
    at net.minecraft.server.v1_5_R3.MinecraftServer.j(MinecraftServer.java:304)
    at net.minecraft.server.v1_5_R3.MinecraftServer.e(MinecraftServer.java:283)
    at net.minecraft.server.v1_5_R3.MinecraftServer.a(MinecraftServer.java:243)
    at net.minecraft.server.v1_5_R3.DedicatedServer.init(DedicatedServer.java:151)
    at net.minecraft.server.v1_5_R3.MinecraftServer.run(MinecraftServer.java:382)
    at net.minecraft.server.v1_5_R3.ThreadServerApplication.run(SourceFile:573)
Caused by: java.lang.ClassNotFoundException: net.minecraft.server.v1_5_R2.Item
    at org.bukkit.plugin.java.PluginClassLoader.findClass0(PluginClassLoader.java:70)
    at org.bukkit.plugin.java.PluginClassLoader.findClass(PluginClassLoader.java:53)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 13 more
2013-09-17 08:43:14 [INFO] [Multiverse-Inventories] Enabling Multiverse-Inventories v2.5-b346
2013-09-17 08:43:14 [INFO] [Multiverse-Inventories 2.5-b346] enabled.
2013-09-17 08:43:14 [INFO] [WormholeXTreme] Enabling WormholeXTreme v1.031
2013-09-17 08:43:14 [INFO] [WormholeXTreme][v1.031] Boot sequence initiated...
2013-09-17 08:43:14 [INFO] [WormholeXTreme][v1.031] Wormhole Worlds support disabled in settings.txt, loading stargates and worlds by our self.
2013-09-17 08:43:14 [INFO] [WormholeXTreme] 49 Wormholes loaded from WormholeDB.
2013-09-17 08:43:14 [INFO] [WormholeXTreme] PermissionAlias backend: 'pex' registered!
2013-09-17 08:43:14 [INFO] [WormholeXTreme] PermissionAlias backend: 'bukkit' registered!
2013-09-17 08:43:14 [INFO] [WormholeXTreme] Attempting to use supported permissions plugin 'PermissionsEx'
2013-09-17 08:43:14 [WARNING] [WormholeXTreme] This plugin is not ready yet.
2013-09-17 08:43:14 [INFO] [WormholeXTreme] Initializing bukkit backend
2013-09-17 08:43:14 [INFO] [WormholeXTreme] Config node PERMISSIONS_BACKEND changed to 'pex'
2013-09-17 08:43:14 [INFO] [WormholeXTreme] Initializing pex backend
2013-09-17 08:43:14 [WARNING] [WormholeXTreme] This plugin is not ready yet.
2013-09-17 08:43:14 [INFO] [WormholeXTreme] Attached to PermissionsEx version 1.19.5
2013-09-17 08:43:14 [INFO] [WormholeXTreme][v1.031] Boot sequence completed
2013-09-17 08:43:14 [INFO] [Multiverse-NetherPortals] Enabling Multiverse-NetherPortals v2.5-b660
2013-09-17 08:43:14 [INFO] [Multiverse-NetherPortals] - Version 2.5-b660 Enabled - By Rigby and fernferret
2013-09-17 08:43:14 [INFO] Server permissions file permissions.yml is empty, ignoring it
2013-09-17 08:43:14 [INFO] Done (4.111s)! For help, type "help" or "?"
2013-09-17 08:43:16 [INFO] /127.0.0.1:46732 lost connection
2013-09-17 08:43:21 [INFO] player1[/192.168.1.11:49962] logged in with entity id 657 at ([mundusprime] 214.27613526949833, 71.0, 536.8997582104788)
2013-09-17 08:43:21 [INFO] Player joined: player1
2013-09-17 08:46:00 [INFO] player1 issued server command: /mvm
2013-09-17 08:46:08 [INFO] player1 issued server command: /mvm set ?
2013-09-17 08:46:24 [INFO] player1 issued server command: /mvm set daytime
2013-09-17 08:46:39 [INFO] player1 issued server command: /mvinfo
2013-09-17 08:46:52 [INFO] player1 issued server command: /mvinfo 2
2013-09-17 08:47:00 [INFO] player1 issued server command: /mvinfo 3
2013-09-17 08:47:08 [INFO] player1 issued server command: /mvinfo 4
2013-09-17 08:48:21 [INFO] give player1 265 5
2013-09-17 08:48:21 [INFO] CONSOLE: Gave player1 some 265 (IRON_INGOT)e[m
2013-09-17 08:48:21 [INFO] points player1 1
2013-09-17 08:48:21 [INFO] e[0;32;1m[RewardMe] e[0;33;22m1 points given!e[m
2013-09-17 08:52:15 [INFO] player1 issued server command: /gamemode 1
2013-09-17 08:52:15 [INFO] player1: Set own game mode to CREATIVE modee[m
2013-09-17 08:53:23 [INFO] player1 issued server command: /gamemode 0
2013-09-17 08:53:23 [INFO] player1: Set own game mode to SURVIVAL modee[m
2013-09-17 08:53:29 [INFO] Connection reset
2013-09-17 08:53:29 [INFO] player1 lost connection: disconnect.quitting
2013-09-17 09:00:01 [INFO] /127.0.0.1:46757 lost connection
2013-09-17 09:00:01 [INFO] CONSOLE: Forcing save..e[m
2013-09-17 09:00:01 [INFO] CONSOLE: Save complete.e[m
2013-09-17 09:00:01 [INFO] CONSOLE: Disabled level saving..e[m
2013-09-17 09:00:19 [INFO] CONSOLE: Enabled level saving..e[m
2013-09-17 09:26:20 [INFO] Disconnecting player2 [/192.168.1.12:50724]: You are not white-listed on this server!
2013-09-17 10:00:01 [INFO] /127.0.0.1:46831 lost connection
2013-09-17 10:00:01 [INFO] /127.0.0.1:46832 lost connection
2013-09-17 10:00:01 [INFO] /127.0.0.1:46833 lost connection
2013-09-17 10:00:01 [INFO] CONSOLE: Forcing save..e[m
2013-09-17 10:00:01 [INFO] CONSOLE: Save complete.e[m
2013-09-17 10:00:01 [INFO] CONSOLE: Disabled level saving..e[m
2013-09-17 10:00:06 [INFO] CONSOLE: Enabled level saving..e[m

The white-list.txt for this sample:

player1

The activity-list.txt for this sample:

You are not white-listed on this server
RewardMe

There are three matches for RewardMe - none including "player1"

$ grep -i rewardme file1 (your log file)
2013-09-17 08:43:10 [INFO] [RewardMe] Loading RewardMe v2.5.1
2013-09-17 08:43:11 [INFO] [RewardMe] Enabling RewardMe v2.5.1
2013-09-17 08:43:11 [INFO] [RewardMe] enabled
$ grep -i "You are not" file1 (your log file

There are no matches for "You are not..."

$ grep -i "you are not" file1
$ (crickets)

The only solution I can think of right now is...

cat white-list.txt >grep-list.txt
cat activity-list.txt >>grep-list.txt
grep -f grep-list.txt server.log >player-activity.log

I can't put the other words in the white-list as it will affect other things.

I wish I could find a solution that didn't require creating a "grep-list.txt" file. Actually, I wish I could find a solution that loaded the "white-list.txt" as variables and allowed me to supply more variables manually for each phrase, then parsed all of server log for those variables (thus eliminating "activity-list.txt" and "grep-list.txt" entirely).

---------- Post updated at 10:34 AM ---------- Previous update was at 10:33 AM ----------

I think perhaps I wasn't clear in my earlier description. I want to locate everything in server.log that is in EITHER the white-list.txt OR the activity-list.txt.

Yes, as I said, if you want to match either the username or the phrase, put them all in the same file.

My understanding is that you wanted to match both the username and the phrase.

Now my current issue with the current solution:

cat white-list.txt >grep-list.txt
cat activity-list.txt >>grep-list.txt
grep -f grep-list.txt server.log >player-activity.log

It appears that the line entry of:

You are not white-listed on this server

is now looking for every word in that phrase and not the phrase. Should I just put it in quotes or do I need to do some other expression formatting?

If you have any blank lines in the white list file, you should remove them.

     -f file, --file=file
             Read one or more newline separated patterns from file.  Empty pattern lines match every input line.  New-
             lines are not considered part of a pattern.  If file is empty, nothing is matched.

Current solution. Added "-w" switch.

cat white-list.txt >grep-list.txt
cat activity-list.txt >>grep-list.txt
grep -f grep-list.txt -w server.log >player-activity.log

This seems to work so far. However, I'm noticing that there are entries in my "server.log" file that have colorized text. These do not show up in results, even if the match on text.