Automate the download of a list of URLs/links

I just wanted to automate the process of downloading a list of URLs/links. I thought about asking a question at SuperUser. So I did that and got a few answers. One of those answers pointed to the piece of software that does exactly what I wanted. The software is a Firefox add-on that I had used a long time ago but couldn’t think about it for this task. Its name is DownThemAll!. Yes with an exclamation mark at the end. I think this has to do with a relief felling you feel when you’re using it.

In this post I’ll walk you through the steps necessary to get your downloads going in a really fast way (fasten your seat belt). No more downloading and saving file by file. Phew!

I’m going to use the same list of URLs I posted on my question at SuperUser.

Suppose you have a list of URLs as this one:

http://creamteam.tv/wp-content/uploads/2010/06/10-Chinatown.mp3
http://mp3muffin.com/train/burbs.mp3
http://www.knoxroad.com/wp-content/music/May/01%20World%20Sick.mp3
http://indierockcafe.com/mp3s/2010/may/5-18/bandofhorses_factory1.mp3
http://www.strangefamousrecords.com/sfr-audio/_common/sagefrancis_slowman.mp3
http://dangerbirdrecords.com/public/theoneamradio/crediblethreats.mp3
http://www.beggarsgroupusa.com/mp3/HolyFuck_LatinAmerica.mp3
http://dl.dropbox.com/u/4746099/05%20Choices%201.mp3
http://subpop-public.s3.amazonaws.com/assets/audio/6801.mp3
http://gvsbchris.com/01%20Sleep%20Paralysist.mp3
http://bantermm.com/tracks/Annuals-Loxtep.mp3
http://downloads.pitchforkmedia.com/The%20National%20-%20Bloodbuzz%20Ohio.mp3
http://stereogum.com/mp3/the%20morning%20benders%20-%20Excuses.mp3

Save this list in a text file [ .txt ] extension and name it MP3s.txt.

Now let’s get DownThemAll! and do the real thing…

1 - Install DownThemAll! add-on for Firefox. After installing restart Firefox.

2 - Go to Firefox menu and select Tools - DownThemAll! Tools - Manager... Right-click inside DownThemAll! screen.

3 - Select context menu Misc - Import from file.

4 - On Import downloads dialog window make sure you select Text Files in the dropdown list.

5 - Point to the text file MP3s.txt that contains the URLs. Click Open button.

6 - On Make your selection dialog window, DownThemAll! will load each link it finds in the text file allowing you to select which ones you want to download and the folder where you want the files to be saved as well.

7 - Make your selections, click Start button.

Here's a screenshot of it just doing the "Downloading" part:

DownThemAll! downloading the list of URLs from my question at SuperUser
Figure 1 - DownThemAll! downloading the list of URLs from my question at SuperUser

Oh… and this is the feature request on DownThemAll! trac opened 3 years ago.

Nice to see this implemented and working so well.

Congrats to DownThemAll! developers and users. Fantastic add-on.

StackOverflow: best place to share/learn programming

I’ve been spending some time of my days helping others at StackOverflow. StackOverflow (SO) is the best place to ask questions related to software programming. There you can be sure that someone somewhere will help you to find an answer to your question.

Note: I just got out of my job at Chemtech. Now I have more time to focus in other things and to think a little bit more about what I want to do next.

As a consequence I decided that I’ll try to give back and share a portion of the knowledge I acquired in these 7 years of programming experience. By means of helping others at StackOverflow I just improve what I think I already know. This is a bit controversial you may say, but I don’t think so. I’m constantly learning/unlearning and discovering new things at SO. I keep trying to sharpen my programming skills. This just happens somewhat in a recursive way. One finding leads to other that then brings you back to the main topic that then expands and so forth.

Give, and it will be given to you. Good measure, pressed down, shaken together, running over, will be put into your lap. For with the measure you use it will be measured back to you.
Luke 6:38

I’m kind of a generalist (I’m after the generalist badge), that is, I don’t tie myself to any technology. I’ll try to use the one that fits better in a given task/job be it close-source or open-source. If a technology allows me to get the thing done that’s the one I’ll choose. This is reflected in the variety of tags at my StackOverflow profile:


Of course there are well established technologies that are easier to work with as is the case of C# programming language, ASP.NET, Java, etc. Again this can be seen both on my tags and in the quantity of questions tagged with such technologies. Being easier to work with means having a greater user base throughout the world and this is reflected at StackOverflow tags as well.

The title of this post is what motivated me to write and I hope will motivate others too so that they give back a portion of what they know.

The point is this: whoever sows sparingly will also reap sparingly, and whoever sows bountifully will also reap bountifully. Each one must give as he has decided in his heart, not reluctantly or under compulsion, for God loves a cheerful giver. And God is able to make all grace abound to you, so that having all sufficiency in all things at all times, you may abound in every good work.
2 Corinthians 9:6-8

I then invite you: if you’d like to get free answers to your coding questions or if you want to write better code or if you want to start at SO and be happy or whatever related to coding problems, I suggest you to read the FAQ and create an account at SO.

You see, we should help each other. There are times that we get stuck at some coding problems for hours and we don’t know what to do to get over them. With millions of programming peers having the most varied backgrounds willing to help and sharing what they know is what makes a developer’s life valuable and exciting. At StackOverflow you find what summarizes this last phrase. In just a minute you can get an answer that’ll allow you to continue your work. If you compare minutes to hours it’s clear why StackOverflow is so fantastic.

The sum of our good programming efforts builds a better software product. LOL <(^^,)> 
By Leniel Macaferi

As of the time of this writing, for stats purposes, I have 4257 reputation,18 badges, 2 questions asked and 288 questions answered at SO. A pretty good amount of answers as a start for someone who’s also after the fanatic badge. I can assure you I’m more than half way there already. Badges are not the point of this post but they make the whole thing a little bit more motivating. Badges at SO are recognitions that come with time just as in real life. The more you share the more recognition you earn.

I take my hat off to the people that started such great site. I hope that it grows even more in the years to come with more and more users trying to learn this admirable profession of software developer that in my humble opinion is the best one. I’m biased towards it after all.

At SO we just happen to have pretty good discussions about anything related to programming including What’s your favorite “programmer” cartoon? and Is a master’s degree overkill? ranging to the more diverse topics. In one of those really interesting discussions I found this brilliant saying that I think holds true in this world:

When you finish your bachelor's, you think you know everything; When you finish your master's, you realize you know nothing; When you finish your doctorate, you realize nobody knows anything, including whether or not you needed to finish a doctorate to realize that.
By Unknown

Although I haven’t taken a master’s or doctorate course yet I can write about the bachelor’s part. When I finished the bachelor’s I already knew that I didn’t know nothing. I think that when I finish the master’s I’ll realize nobody knows anything. One less step to realize that. What will happen then if I take a doctorate course? Simply put: (no one knows nothing ) ^ Googol I think. :)

Final Note: I also put in practice this same share and learn principle at ProZ.com.

Further references
I suggest you check these two posts by Jon Skeet:

Writing the perfect question
Answering technical questions helpfully

Manipulating Properties and Methods with Reflection

Today I’m going to write about a simple yet powerful way to manipulate properties and methods of a given type. We’ll set values to properties and invoke methods through the use of reflection.

Maybe you haven’t had the chance to use reflection or maybe you even don’t know what is reflection.

Reflection is one of the most powerful features a programming language has as is the case of C# programming language that I’ll be using in this post.

In just one phrase extracted from Wikipedia we can define what is reflection:

Reflection is the process by which a computer program can observe and modify its own structure and behavior. The programming paradigm driven by reflection is called reflective programming.

Our objective is to manipulate the property value and methods of a type by means of reflection, that is, we’ll have access to these properties and methods by looking at the type’s metadata.

Let’s exemplify with a specific case that I came into while I was working on my first project at Chemtech: we had our ASP.NET solution divided into specific projects in what we call the MVP pattern. I suggest that you to read the post Model View Presenter pattern with Castle in ASP.NET to get a grasp of it.

In a given task I had 15 static textboxes inside an .aspx page (View) and I needed to set their values inside the Presenter. What is the best way to do it? Should you hardcode each and every textbox setting their values? It’s not an elegant solution. So how to get over it? Using reflection to get the type’s metadata. Let’s see how it is easy.

Let’s get to the code:

private void BindProperty(string viewPropertyName, object propertyValue)
{
    // Getting the property I want to use.
    PropertyInfo propertyInfo = typeof(IMyView).GetProperty(viewPropertyName);

    // Verifying if the property was acquired with success.
    if (propertyInfo != null && propertyInfo.CanWrite)
    {
        // Set the property value.
        propertyInfo.SetValue(view, propertyValue, null);
    }
}

Reflection allows us to make a generic piece of code, that is, with only one method we can set the value of all the textboxes.

The above method receives the property name as a parameter and we also pass to the method the value we want to assign to the property.

This same approach can be used with methods. Let’s see an example:

private void BindMethod(string viewMethodName, object[] methodParameters)
{
    // Getting the method I want to use.
    MethodInfo methodInfo = typeof(IMyView).GetMethod(viewMethodName);

    if (methodInfo !=null)
    {
// Call/Invoke the method with the desired parameters. methodInfo.Invoke(view, methodParameters); } }

Again we have a generic piece of code that can be used to call whatever method we want passing whatever parameters we want.

To use PropertyInfo and MethodInfo we declare the namespace System.Reflection.

We could make these methods even more generic by passing the view object (type) we want as a parameter. Bellow I show a class named ReflectionUtil with such modifications:

/// <summary>
/// Utility class for Reflection operations.
/// </summary>
public class ReflectionUtil
{
    /// <summary>
    /// Sets a value to a property through the use of Reflection.
    /// </summary>
    /// <param name="obj">Object that owns the property</param>
    /// <param name="propertyName">Property name</param>
    /// <param name="propertyValue">Value to be set</param>
    public static void BindProperty(object obj, string propertyName, object propertyValue)
    {
        // Getting the property I want to use.
        PropertyInfo propertyInfo = obj.GetType().GetProperty(propertyName);

        // Verifying if the property was acquired with success.
        if (propertyInfo != null && propertyInfo.CanWrite)
        {
            // Set the property value.
            propertyInfo.SetValue(obj, propertyValue, null);
        }
    }

    /// <summary>
    /// Calls a method through the use of Reflection.
    /// </summary>
    /// <param name="obj">Object that owns the method</param>
    /// <param name="methodName">Method name</param>
    /// <param name="methodParameters">Method parameters</param>
    public static void BindMethod(object obj, string methodName, object[] methodParameters)
    {
        // Getting the method I want to use.
        MethodInfo methodInfo = obj.GetType().GetMethod(methodName);

        if (methodInfo != null)
        {
// Call/Invoke the method with the desired parameters. methodInfo.Invoke(obj, methodParameters); } } }

OK. Now that we have the methods defined, let’s use them.

I’ll use the application I’ve shown in Model View Presenter pattern with Castle in ASP.NET as the base for this post.

IMyView interface declares the following members:

public interface IMyView : IBaseView
{
    event EventHandler FirstLoading;

    void MyMethod();

    string TextBox1Text { set; }
    string TextBox2Text { set; }
    string TextBox3Text { set; }
    string TextBox4Text { set; }
    string TextBox5Text { set; }
    string TextBox6Text { set; }
    string TextBox7Text { set; }
    string TextBox8Text { set; }
    string TextBox9Text { set; }
    string TextBox10Text { set; }

    void FillGridView1(List<object> values);
    void FillGridView2(List<object> values);
}

MyView implements what’s in IMyView interface:

public partial class MyView : Page, IMyView
{
    private MyPresenter presenter;

    .
.
.
#region Implementation of IMyView public event EventHandler FirstLoading; public void MyMethod() { Response.Write("Reflection power"); } #region TextBoxes public string TextBox1Text { set { TextBox1.Text = value; } } public string TextBox2Text { set { TextBox2.Text = value; } }

.
.
. #endregion #region Methods public void FillGridView1(List<object> values) { GridView1.DataSource = values; GridView1.DataBind(); } public void FillGridView2(List<object> values) { GridView2.DataSource = values; GridView2.DataBind(); } #endregion #endregion }

Inside the Presenter MyPresenter I implement the logic with the following:

/// <summary>
/// Specific FirstLoading implemented by each inheritor
/// </summary>
protected override void FirstLoading()
{
    view.MyMethod();

    // Useful technique to avoid calling 10 lines of code.
    for (int i = 1; i <= 10; i++)
    {
        ReflectionUtil.BindProperty(view, string.Format("TextBox{0}Text", i), (i * i).ToString());
    }

    // Useful technique to avoid writing methods almost identical just to change a method call.
    for (int i = 1; i <= 2; i++)
    {
        // Do some common logic here...

        // Call a specific method with Reflection.
ReflectionUtil.BindMethod(view, string.Format("FillGridView{0}", i), new object[] { new List<object> { i * 2, i * 3, i * 4, i * 5 } }); // Do some common logic here... } }

When we run the web app, we get the following result:

Reflection result

I covered just a simple use case of reflection in this post but it’s useful in our day to day job.

The technique implemented in this post allows us to write clean and generic code. I have already used it in another project which allowed me to write as few lines of code as possible. Let’s say it decreased from 250 sloc to 25 sloc. A reduction of 10 times sloc.

Last note
I read the article Survival of the Fittest: Natural Selection with Windows Forms on MSDN Magazine when I was on the 9th period of the Computer Engineering course studying Artificial Intelligence. From this moment on I started to visualize the importance of reflection and how powerful it is. The referenced article discusses about Genetic Algorithms and the author’s sample code makes extensive use of reflection. Great read if you’re interested enough! Oh, I even translated this article to Portuguese at that time. I want to put it online so that others can benefit from it. :o)

Visual Studio 2010 C# ASP.NET Web Application
You can get the Microsoft Visual Studio Project at:

http://sites.google.com/site/leniel/blog/ReflectionWebApp.zip

To try out the code you can use the free Microsoft Visual Web Developer 2010 Express Edition that you can get at: http://www.microsoft.com/express/Web/