I would like to clean up a bunch of broken links on our site by removing them programmatically. (There's 861 of them.) All of the links look like this:
I'm dealing with strings with unpredicted and variable amounts of characters like this.
Then, what I want to do is remove the entire string from the file except the string between the closing and opening arrows (in this particular case Bolter #3548), except I should probably enclose it in paragraph tags.
I have no clue how to do this. A combination of sed and split? Awk? Grep? Perl?
I have a feeling that no matter what I do I'm going to end up breaking a bunch of pages, but editing 861 files individually doesn't exactly thrill me.
Are these pages static or dynamic? If dynamic, you'll need to address this issue with changes to whatever generates these pages.
I don't see the mentioned "string" to be removed (see above) in the original sample file.
What makes the "link" invalid to be removed? u=xxxxxxxx value? A certain invalid pattern in the link construct? Something else? How to distinguish a "bad" link from the "good" one?
Please provide a small representative data input and the corresponding desired output. Don't say "probably", "maybe" etc. - state exactly what you want as the result based on your SMALL and REPRESENTATIVE input.
However, there is not a one-to-one correspondence between the Bolter # on the old board and the Bolter # on the new board, so we can't just edit the links. The best solution I can come up with is one of two things. Either remove the link and text completely, or leave the text behind but remove the link.
So, to use the example that I posted originally, I would want to go from this:
Now that I think about it, it might be better to remove the link AND the text, since the Bolter # is likely to be wrong anyway. And that might be easier to accomplish anyway.
I'm not sure about a 'shell' solution in bash or whatever shell you use, but I've done something similar using Perl. I had to change the title of a few hundred web pages and used the perl inline-editing feature. Might be worth while looking at.
@trudge,
' but I've done something similar using Perl. I had to change the title of a few hundred web pages and used the perl inline-editing feature. Might be worth while looking at.
how about showing those/that - saying you've '...done something similar...' without showing is not helpful its just noise
Perl can be run with certain arguments on the command line to make it do different things. One of things it can do is called 'inline file editing'. What that means is that Perl can open a file, do some editing, and close the file - all without an editor of any kind.
In the directory with all my HTML files I ran this 1-liner:
You might recognize part of that code to do some string substitution.
Basically it tells Perl to walk through all '.html' files in the current directory, copy them to a new file with a '.old' extension, replace 'Tutorials' with 'Refreshments', and do this for each file.
After I checked to see everything was fine (all 200-some files), I deleted all the files ending in '.old'
Thank you for that. I will study it and test it out on a single file. I would assume that, if I decide to just remove the entire string, I would simply make the second expression empty.
Yes, substitution with nothing is a deletion.
As was already suggested, you can directly modify the input files by option -i.old (backup the original as .old , just -i is no backup).
After talking with the owners, they indicated that they wanted to change the old url to the new url but retain the data. So, what that would mean, in practical terms is that this line in the file:
It didn't change a thing. It did create a .old file that diff says is identical to the original file. I created the regex using regexr, and it matched as I expected, but apparently it doesn't match the actual file?
If I put this into a perl script, would that solve the problem?
mod_rewrite is based on a PCRE regular-expression parser, so when you use mod_rewrite you can test and validate everything.
If then you later decide to change all the URLs by editing them, and you don't care about the SEO penalty this will incur, then you can use the the mod_rewrite rules you developed in scripts to create all your edit changes on your site site before cutting over and going live with changes.
Honestly, in doing this kind of work for as long as I can remember, I have always used mod_rewrite and have rarely run any script to make changes.
I think this is how most people do it; because it is easier and preserves SEO for all links, as most people care about SEO.
s command missing?
Also, you can add a \b "word boundary" anchor to increase precision. .* is greedy, .*u= finds the rightmost u=. While .*? finds the minimum(nearest) match.
The g modifier (at the end of the s command) applies the whole substitution again on the remainder of the line. Would find another following URL
Now I'm trying to get it working, but I've run into a very odd problem. None of my page loads are showing up in the apache logs - none - not in the access log. Not in the error log.
I enabled rewrite logging at the trace level, and I can see the rule being compared to pages that don't match. The problem I have now is that apache is not logging any traffic from my IP (and from a lot of other iPs), and I don't know why. But, without log entries, it's really hard to troubleshoot.
This is the contents of the .htaccess file in document root.
RewriteEngine on
RewriteRule ^/bboard.*u=(\d){8} /ubbthreads/ubbthreads.php?ubb=showprofile&User=$1 [NC]
But the rule is not working. It appears that mod_rewrite is trying to apply the rule but it's not applying it to the actual urls that have the matching string in them.
This is the error I get when I click on one of those bad links.
The requested URL /bboard/cgi-bin/ultimatebb.cgi was not found on this server.
Is there a reason that everything from the question mark on is ignored in this error message? Do I need to be using QUERY_STRING instead of the regex I'm using? And if I do, how do I capture the stuff after u=?
