I have a little script that I got of the internet and that I use in Squid to block ads.
I used that script with linux but now i have moved my servers to freebsd. I have a step learning curve there but it is fun: Back to the script issue.
The script used to work i with linux but freebsd is a bit different.
This line is causing me issue
If I use the line above it in the script below, the destination folder is going to be completely emptied. The goal is to get rid of the HTML tags "(^|\.)" in the list that is given by http address "pgl.yoyo.org" for bad ad website. Then it is used by squid proxy.
The line above is unusable. The script works well if i modified the line above without the pipe, grep and the tags
Then the list is being updated correctly and not emptied but still with the HTML tags in it.
#!/bin/sh
# Get new ad server list
/usr/local/bin/wget -O /tmp/temp_ad_file \
'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=squid-dstdom-regex;showintro=0&mimetype=plaintext'
# Clean HTML headers out of the list
cat /tmp/temp_ad_file > "/usr/local/etc/squid/squid.adservers"
# cat /tmp/temp_ad_file | grep "(^|\.)" > "/usr/local/etc/squid/squid.adservers"
# Refresh Squid
/usr/local/sbin/squid -k reconfigure
# Remove tmp file
rm -rf /tmp/temp_ad_file
ahamed101, thanks for replying.
I have pasted the start of the file (txt file)
The HTML tags would be "(^|\.)" & $. If left in that list, the acl squid can't use the file.
My first step is to go and download the list as a txt file and save it to squid folder as squid.adservers.
Then the line above is to update the list once every 3 days. With the line above the destination folder squid.adservers gets emptied when the script is run and the acl inside squid is then complaining.
I am confused... You have pasted the contents of /tmp/temp_ad_file right?... The grep statement is looking for pattern (^|\.) and populating those in the squid.adservers file... try the grep statement without the redirection and see if you get anything on the screen...
Sorry for the confusion ahamed101
tried without the redirection so staying in the tmp directory with folder "temp_ad_file"
grep '(^|\\.)' /tmp/temp_ad_file. When I look in the temp_ad_file and the list is complete with '(^|\\.)'
Here is an extract below.
Basically what I am trying is clean the html part and redirect the entire list to the squid folder to file called squid.adservers
A couple of observations that I have that might help clear the problem. First, the original post refers to 'removing html' from the file. However the file pulled from yoyo.org with wget using text/plain does not contain any html. More so, the reference to
seems to indicate that while incorrectly calling (^|\.) HTML, these strings are not desired. Depending on how squid is configured, this is true, they need to be removed.
The file from yoyo.com is a list of regular expressions and if squid isn't configured with acl ads dstdom_regex -i "[/usr/local]/etc/squid.adservers" then the regex parts will cause problems. I believe this is the reason things have stopped working is because the configuration on the old machine isn't the same as on FreeBSD. Given this, the original code that is extracting the regex lines from the yoyo.com data using the regex string makes sense.
This doesn't explain why the output file is ending up empty, but might change the focus on the problem a bit. If the squid config is changed to match the old machine, then the regex file can be used as is, otherwise the regex portions should be stripped:
sed 's/[()|.$^]//g' /tmp/temp_ad_file >/usr/local/etc/squid/squid.adservers
Care should be taken if these strings are used without the regex as they might match more URLs than desired.
I'm interested in knowing if the sed above has the same problem -- generates an empty file. If it does, then I question the permissions on the output file. What happens if the output file is changed to something like >/tmp/foo ?
agama, thanks for your help here. You have understood my issue better than I could ever explain it. I am very sorry for calling those tags HTML when they were not and causing so much confusion. My apologies.
Coming back to the script; the tags are completely gone now. My ACL in squid is back to working and the final destination folder does not get emptied any more. Thank you so much for your help again agama.
One small thing still where I am having some difficulties; learning curve is step:)
When the script is run, the /tmp/temp_ad_file is being displayed in the console. Is there a way to not display the temp_ad_file in the console ?
#!/bin/sh
# Get new ad server list
/usr/local/bin/wget -O /tmp/temp_ad_file \
'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=squid-dstdom-regex;showintro=0&mimetype=plai
# Removing repeated lines
cat /tmp/temp_ad_file | uniq
# Removing blank lines
sed /^$/d /tmp/temp_ad_file
# Cleaning list
sed 's/[()|.$^]//g' /tmp/temp_ad_file > /usr/local/etc/squid/squid.adservers
# Refresh Squid
/usr/local/sbin/squid -k reconfigure
# Remove tmp file
rm -rf /tmp/temp_ad_file
Kind Regards,
---------- Post updated at 05:18 PM ---------- Previous update was at 04:53 PM ----------
I have bought this book "mastering.unix.shell.scripting" to try to understand a bit more about nix scripting. I would recommend this book for advanced users: For someone like me, its a bit tough.
Glad you've got it working, and it seems you're willing to understand the why with the how which is important.
In general, if you aren't doing anything with the output of the above commands, then you don't need to run them; just comment them out, or remove them completely. However, looking at your script, I believe the intent was to execute these as a pipeline rather than by themselves. The data from yoyo.com needs to be sorted for uniq to work, and since you get the input from an external source (order unknown), best to sort with the unique option rather than assuming it's sorted. The sed to remove blank lines can be combined with the sed to delete the regex stuff, so the pipeline is just two commands to accomplish all 4 things:
agama, thanks
I have amended as per your suggestions
I get a better result even
I have moved up the "cleaning list" line so I have it above the "uniq" variable. That way it is sorted, cleaned, blank lines removed, and then all repetition being removed.
I am using the "uniq" filter on the final destination folder which is squid.adservers and not anymore on the /tmp/temp_ad_file.
danmero, thanks for helping as well.
Kind Regards,
#!/bin/sh
# Get new ad server list
/usr/local/bin/wget -O /tmp/temp_ad_file \
'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=squid-dstdom-regex;showintro=0&mime
# Cleaning list
sort -u /tmp/temp_ad_file | sed '/^$/d; s/[()|.$^]//g' > /usr/local/etc/squid/squid.adservers
# Removing repeated lines
cat /usr/local/etc/squid/squid.adservers | uniq
# Refresh Squid
/usr/local/sbin/squid -k reconfigure
# Remove tmp file
rm -rf /tmp/temp_ad_file
The sort -u already makes the records in the file unique as it sorts them. I didn't explain before that the -u option stands for unique. Further, your output isn't going anywhere except standard output, so executing the command isn't doing anything to the file.
