Thursday, September 12, 2013

Smarter playlist length calculations

For my playlist controller/generator application, I wanted to have a way to produce an accurate list length to reflect about one day's worth of songs. I figured that if I knew how long every track was in my library, I should be able to get a good estimate as to how many tracks I could potentially play in a single day. From that, I then can figure out how many tracks to load into the playlist. So, first I need to have the lengths, in seconds (and fractions of seconds) for each and every track in the library.

I have my library partitioned into different categories, one category per track. I did not want to use the genre for this, as the genre could be something that does not reflect my own personal preference for categorizing my music. I have the categories organized in my settings file so that they are listed in order of my preference. This is done in a two-dimensional list configuration in python. The placement order in this list determines the "weight" of a particular category, which effectively determines how much more or less I'd like to hear songs in that category.

I divide up the tracks based on category, and take the geometric mean of the lengths of each track in each category. For those who are unfamiliar with a geometric mean, it is another kind of average that can be taken. The more common arithmetic mean is you sum up the values, then divide the sum by the number of values present. The geometric mean is taken by multiplying all of the values, then take the Nth root of the product, where N is the total number of values. So, if you have three values, 3, 2, 4, the arithmetic mean of these would be (3 + 2 + 4) / 3 = 3. Th geometric mean would be the cube root of (3 * 2 * 4), or the cube root of 24, which would be 2.88... This kind of average tends to predict lower than the arithmetic mean, which in this particular scenario would be like assuming what might be possible if a lot of smaller-length tracks were played within a single day.

Unfortunately, the calculation for geometric means results in insanely huge numbers due to multiplication, and actually can end up going out of bounds for floating-point numbers due to the size of the number that is calculated. So, there is a trick that can be done to avoid this. The trick is to use the arithmetic mean, and the harmonic mean. The harmonic mean is summing the reciprocals of the values, and then multiplying that by the total number of values (in the above example, you would do 3 * (1/3 + 1/4 + 1/2) = 2.77...). then you do the geometric mean of the arithmetic and harmonic means (square root of 3 * 2.77... = 2.88...).

Because I process favorites vs non-favorites differently, I run this geometric mean on all tracks in a category, and then just the favorites in that category, and apply the weights I've defined in my settings file for favorites or normal tracks to the results, and sum the two together to get the adjusted geometric mean for the whole category.

I then do a weighted average for the adjusted geometric mean for each category, based on the weights I've calculated earlier. A weighted average is to multiply the category's geometric mean by the category's weight, and summing that with the product of each category, and dividing it by the sum of the weights. The end result is a very good estimate for the average length, in seconds, for each track. I divide the number of seconds in a day by this average length, and I get the total number of tracks per day, which will be the basis for the number of tracks I load into the playlist. The end result is that different tracks impact this calculation in different weights based on the category and whether it is a favorite track or not. My favorites in my favorite category have a bigger impact than a non-favorite track in my least favorite category. Smaller-length tracks have a bigger impact on this number than longer tracks.

Tuesday, September 10, 2013

Making my playlist processing be concurrent.

This may come as a shock to some people, but I have this itch to continually tweak my music playlist. It started with me having a set of songs I liked... And [un?]fortunately, this collection grew, and grew. Soon, I wanted to have a way to organize my music and ensure that I'm listening to a certain set of songs. Enter the iTunes Smart Playlists, which let me define, in Set Theory-like notation, multiple playlists each following a different set of rules, that I could combine together and randomize to produce a "daily playlist", which I loaded into my Gen 1 iPhone. This worked fairly well.

Well... eventually I replaced that phone, and chose a less technologically-advanced phone (read: not a smart phone), and had to come up with some other way to get my music. So I took my music, and used a plethora of different streaming servers, building the same kind of functionality as I had with the iTunes smart playlists, building a daily playlist to stream to me at work (or wherever I can get the stream from). I eventually got fed up with most of the streaming servers, and settled onto Music Player Daemon (MPD). I also found a nice python library to interface with the MPD server, and control it (python-mpd2). This meant I could basically write my own playlist generator, how I wanted it, and have my music streamed through MPD, without all of the annoyances that the proprietary streaming servers were doing.

Now, several generations and revisions later, I have my own music database (the backend is unimporant, it will work with any database backend that SQLAlchemy can use) for my playlist generator, which has transformed into a playlist controller (it continually runs, and updates as songs are played through the day, eventually loading in a new batch of songs as the internal playlist gets used up). I have rebuilt it so that the database updating, and the playlist generating is all done concurrently, with multiple "worker processes" that drive handles chunks of the processing of each independently of the other processes. The beauty of this is that I can leverage multi-core computers for the playlist controller, and be able to have the processes spread around the different cores, which gives a nice performance boost.

Now, I am trying to add in new features to it. I want it to scrobble to last.fm. I want my holiday music playlist modifications to work again. I want it to play stuff from podcasts, I want to be able to mix in music from Pandora. These will keep m busy, now that I think I have the core processing working in a way I am comfortable with.

Thursday, May 23, 2013

A Christian's Response To: A Christian's View of World of Warcraft (published in 2008)

I found this little jewel out on the 'net, and feel a need to respond to the blatant misconstruing of what is or isn't OK with World of Warcraft. I am a Christian (Confessional Lutheran, to be precise), and I also enjoyed playing World of Warcraft. I don't like it when people muddy the name "Christian" with stuff that is not Christian, and feel compelled by my Christian Freedom and Christian Responsibility to respond to this, even if I am about 5 years late on finding it. I am not a fan of the Legalism or the Moralism that plagues American Protestantism. So... here we go:

Evil Character and Quests
You begin the game by creating a character choosing from ten races and nine classes. The classes in World of Warcraft areWarriors, Mages, Druids, Hunters, Warlocks, Priests, Paladins, Rogues, and Shaman. Races are divided by Horde or the Alliance. The Alliance includes Humans, Night Elves, Dwarfs, Gnomes and Draenei. The Horde includes Orcs, Tauren, Undead, Trolls and Blood Elves. Male and female sexes are available in all races.
The classic argument about playing "evil characters" in a game. This has recently been touched upon quite well by Rev. Jonathan Fisk at Worldview Everlasting. This is basically an arbitrary line in the sand being drawn. Why do I say that? Because in the game, you pretend to be a different person, basically, you lie. So this line is drawn claiming that one kind of lying (role-playing a good character) is better than another kind of lying (role-playing a bad character). Further, this argument could be applied to condemn any writer to ever have evil characters portrayed in their books. Because they have to get into the role of those characters to know how the characters would act/react in the book. Or, for that matter, you better just not read any book with conflicts between good and evil characters within it, because the author was obviously being evil when writing about the evil characters, and it is better to avoid it entirely. Oh, wait, this means you better skip reading the Bible, because it is filled with all kinds of conflicts between good and evil, most notably between the triune God and sin, death, and the Devil. Like I said, it is an arbitrary line... and one that is not made in the Bible at all. As it is not expressly forbidden in the Bible, it becomes a matter of Christian Freedom. The correct term for it is "adiophra", which means those things not expressly forbidden in the Bible.

Clinical psychologist and assistant professor at Harvard Medical School, Dr. Maressa Orzack believes 40% of  World of Warcraft subscribers are addicted to the game. Said the doctor: “I think there needs to be warning labels on MMORPGs like World of Warcraft, similar to warning labels on cigarettes.
Sorry, but psychology is not a study based on the Bible, or even remotely connected to the Bible (and in fact, is arguably contrary to the Bible on many points). This has absolutely nothing to do with Christianity, and everything to do with human reason. The jury is still out as far as gaming "addiction" is concerned, and I don't see this as any kind of point with merit when defining Christian reasons to oppose playing WoW.

Characters take on “quests” or assignments that include traveling through terrains and fighting different kinds of monsters. Players climb and improve skill as they perform the quests. At times a player may need the help of other players. These are called dungeons which can be two to five players or groups of 30 to 40. It can take several weeks or months in order to pass the dungeon depending on the skill and the playing style of the player.
So what? It is, by definition a game, with rules connected to how to play the game. A game without rules would be unplayable, I don't care if we're talking a video game, a board game, or a sport. If you are complaining about how much time is being used in playing the game, then just control how much time is spent playing it. I played WoW a long time, and rarely did raids or even dungeons. I could drop in and out of the game easily, and had a number of characters at maximum level. Beyond that, I am lost as to how this is a Christian "red flag". I totally call this adiophra.
Many of the quests involve violence. Much of the game is spent fighting a diverse and sometimes frightening assortment of computer-controlled creatures gone bad-wild boars, wolves, rogues, undead creatures, dragons, thieves, etc. that include graphic depictions of blood during combat with appropriate sound effects.
Sorry, but the Bible is not against violence. Nope. Don't believe me? Check out the book of Joshua. Or possibly 1 & 2 Chronicles, 1 & 2 Kings, or really the vast majority of the Old Testament. Or even the Gospels for that matter (especially the part where Jesus was beaten, bruised, and killed for all of us... kinda an important facet of Christianity). And the Bible especially is not against simulated violence. Sorry, adiophra.

Players control an avatar (character) in the world inside the game. They explore the landscape, perform all activities and perfect skills while winning items. Items include things like herbs, manna, and gold. These items, along with different weapons and armor, are a focus of the game as they improve various attack or defense skills.
Like I said, it is a game. I fail to see, entirely, how this is somehow wrong. Arbitrary line anyone? Gonna start banning reading books of fiction? Or what about watching TV or movies? Adiophra.

Casting Spells, Witchcraft, Ghosts, Violence
Players fight by casting spells. More correctly – they pretend to cast spells.  Different spells have different effects on other players or monsters. Players earn the ability to cast spells by visiting trainers and by receiving items dropped by monsters, found through quests, or sold by other players. The strength of the spells depends on the players’ level and earned items. Spells can improve or weaken friends and enemy troops. Other spells do direct damage to enemies or heal allies.
Yep. the key word is pretend to cast spells. It definitely is nothing like the real thing. The "spells" change certain values contained within an internal database containing numeric representations of hit points or other statistics. Absolutely none of it is even remotely witchcraft. Any more than playing Dungeons & Dragons, or Magic: The Gathering, or even just playing a game of make-believe Harry Potter. Sheesh. Get a grip on reality, would you? This is nothing more than adiophra.

When a player dies, they receive a dialog box to “Release Spirit.” By pressing this button their character is transported to the nearest graveyard as a ghost. Priests, shamans, paladins, druids, and warlocks all have the power to bring other characters back from the dead. The method and frequency with which they can do so varies by class.
Because this kind of thing happens in real life and has real implications in the real world. I'm sorry, where's the plain Bible passage that states doing this in a computer game is expressly forbidden? Anyone? Anyone? Bueller? Adiophra

World of Warcraft is rated “T” for Teen with warnings of blood, suggestive themes, use of alcohol, and violence. WoW is a place where thousands of people communicate in an unmonitored environment. Foul language is used consistently. Alcohol is occasionally referenced. Players can buy and consume alcohol in taverns and pubs to the point of getting drunk.
Point to me the passage that says rated "T" for Teen video games are evil. I can see the argument that these ratings are good tools for parents to be able to decide on their own whether it is appropriate for their children or not, but it definitely is not anything to do with Christianity. Adiophra.

Women characters are dressed scantly. Promoting lust. Players can “flirt” with one another while some females are scantily clad. (A teenage girl suggested I add this red flag, her words dressed whorish and horrible!)
Do you think that by avoiding seeing a scantily-dressed woman, you will somehow not be sinning against the 7th commandment? We all break all of the commandments in all of our thoughts, words, and deeds. We are accountable for it all, save for the gift of salvation we received in Christ, and the gift of faith we have been given through the Holy Spirit. Even so, we still regularly need to seek forgiveness for breaking all of the commandments, and staying away from a game that has scantily-dressed women characters because it promotes lust is really not going to make you somehow be better than anyone else, or somehow avoid sinning. While I personally wish the games had more respect for the female body than is often shown, at least in World of Warcraft, the most you will see is nothing more than you would see going to a beach. I wouldn't necessarily call this adiophra, but I will say that it is rediculously legalistic and misses the point of Christianity by emphasizing the wrong thing -- what we can do to avoid sin, rather than repenting and receiving forgiveness for our sins daily.

God says “Abstain from all appearances of evil. I didn’t write it God did. 
Ok, I believe you are quoting 1 Thess. 5:22 (KJV) here. Let's see if this one verse really means what you are saying (or rather, implying), or if it means something else. There are three main rules of properly understanding the Bible: context, context, and context. So, let's look at the immediate context surrounding this one verse. I will use the ESV as my translation of choice, because, to be honest, I don't think in Middle English, and in fact, modern people in general don't. 1 Thess. 5:12-22 (ESV):

12 We ask you, brothers, to respect those who labor among you and are over you in the Lord and admonish you, 13 and to esteem them very highly in love because of their work. Be at peace among yourselves. 14 And we urge you, brothers, admonish the idle,[c] encourage the fainthearted, help the weak, be patient with them all. 15 See that no one repays anyone evil for evil, but always seek to do good to one another and to everyone. 16 Rejoice always, 17 pray without ceasing, 18 give thanks in all circumstances; for this is the will of God in Christ Jesus for you. 19 Do not quench the Spirit. 20 Do not despise prophecies, 21 but test everything; hold fast what is good. 22 Abstain from every form of evil.
Well, it appears to me as though this is the end of a letter. In fact, it is the end of Paul's first letter to the Thessalonians. It appears to be some parting advice, that builds upon points made through the entire book. It looks like Paul is trying to tell the congregation in Thessalonica to be supportive of each other, to respect each other, and to build each other up. Also to test everything -- in the context provided, it appears any prophecies (which in the context means teachings people have claimed are from God Himself) they have received -- and keep the ones that are true, but to reject the ones that are false. It is an admonishment to avoid false teaching (prophecies), and the way to determine the false teachings from the true ones is to test the teachings against God's Word. So... I'd have to say that this verse really is completely out of context & off-topic with the idea of playing a character in World of Warcraft, even an evil character. Sorry, doesn't work.

The Bible is clear that casting spells is an abomination to God. Those who practice witchcraft (sorcery) will not inherit the kingdom of God (Galatians 5:20-21). These practices are anti-God and are in rebellion against Him. Also see Deuteronomy 18:9-14; Isaiah 44:25; Jeremiah 27:9; 2 Kings 21:6; 23:24, Ezekiel 21:21; Isaiah 19:3; 1 Samuel 28, 1 Chronicles 10:13-14.
Ok. So this is about real-life actually performing witchcraft. It is an abomination because it puts the power of humanity first, and God is unnecessary. This would be what you would see in pagan ceremonies, and it doesn't jive with the Bible. However, is this really what is done in World of Warcraft? Nope. What is done is a mathematical equation that produces a change in numerical values contained in a database. All with some spiffy graphical effects. What's the difference in WoW between casting a fire bolt or hitting with a sword? Underneath the surface, in the programming world that is what drives WoW itself... really not much. They are basically the same thing. Everything has been abstracted so much that it is truly harmless. Me making my Warlock cast Siphon Life on a target in the game is not going to teach me how to cast a real-life spell that can somehow do the same thing. The "magic" in World of Warcraft is just a numbers game, nothing more. It is no more problematic for a Christian than playing Monopoly or Yahtzee. Oh, not a big fan of the NLT, let's try looking at a better translation, like the ESV. If needs be, I can debate this point further, but I'd recommend you check out Worldview Everlasting, and find the videos there that touch on this very point (Pr. Fisk himself plays Magic: The Gathering, and taught it to his kids.... and he makes the point as to why it isn't the same thing as actual, real-life heebee-jeebee magic pretty clear).


What Does God say about the Appearance of Evil?
Various message boards have threads of discussion about Christians playing WoW. Players say they even use the game to be a witness and lead other to Christ. They defend the right to play this game because it is not really evil but only a game.
OK, its only a game–Lets say it is not evil for arguments sake. Does it appear evil?
Is it OK to play a game about casting spells? Gamers aren’t really casting spells. They are only humans, not priest or warlocks, and they only direct the character that they have chosen to go through an adventure casting spells.
Pretending to cast spells is not the same as casting spells but I don’t want my children to pretend to cast spells.
These are my children and this is my decision. Why are people so angry (deleted over 200 F…U. comments so far)  for stopping MY children from playing this game?
I already answered this one. You are pulling that verse out of context, claiming it is what God says. However, when tested against the Word of God, in context, it is found to be a false interpretation. So, according to the very passage you kindly put up there, I will abstain from this evil twisting of God's Word. And will call it for what it is openly.

The Real Spiritual Battle
We are in a battle daily–not a virtual battle–a real battle! The Christian walk is a battle against our sin nature or our flesh. “The flesh” refers to the fallen nature still with the believer. In Romans 6, Paul says the old man has been crucified and that we can overcome the flesh by reckoning ourselves dead to sin and by yielding ourselves to God. God’s Holy Spirit assists us in our battle against the flesh.
God wants to transform us into Christ’s likeness (2 Cor. 3:18; Rom. 8:29 and 12:1–2). If we live in the Spirit let us also walk in the Spirit (allowing the Spirit to control our lives). Walking in the Spirit is the daily experience of the believer who feeds on the Word, prays, and obeys what the Bible says.
Legalistic view of the passages. Completely confusing Law and Gospel. As a Lutheran, I confess that the Book of Concord summarizes the doctrines found within the Bible. As such, within the Book of Concord, the three purposes of the Law are defined as:


  1. that "thereby outward discipline might be maintained against wild, disobedient men [and that wild and intractable men might be restrained, as though by certain bars]"
  2. that "men thereby may be led to the knowledge of their sins"
  3. that "after they are regenerate. . .they might. . .have a fixed rule according to which they are to regulate and direct their whole life"

Another way to put it is: 1. as a Curb, 2. as a Mirror, and 3. as a Rule. The primary purpose of the Law is to Show us our Sin (#2, a Mirror). It also is to help curb us from doing something we shouldn't (like run around killing people in real life). The 3rd use is interesting, because when properly taught Law and Gospel, it is automatically done thanks to the regeneration of the Gospel, and is not something we have to make ourselves do, but will automatically done. The "Walking in the Spirit" is that 3rd use of the Law. When shown our sin, and we repent of it, then receive the forgiveness of our sins (through absolution, baptism, and Holy Communion), we will do good works, and live good lives. Not because we must do it in order to be good Christians, but because we have been regenerated in Christ, and the good works are the fruits of this regeneration. But you are right, we are constantly in a spiritual battle -- between the "old Adam" of our sinful natures, and the "new Adam" that is the result of Christ's salvation and the faith brought to us as a free gift from the Holy Spirit. We are simultaneously sinners and saints. And we daily need to kill and bury the old Adam through repentance and regenerate the new Adam through the gift of forgiveness we receive. We need to have a broken and contrite heart. The Law kills the old Adam, and the Gospel regenerates the new Adam. This is how we live as Christians. Not by trying to follow a set of man-made rules trying to be moralistic and show everyone how good we are.

How We Spend our Time
Even secular parents voice concerns over the addictiveness of the game, and the creation of a sedentary, non-communicative, non-social existence. Wikipedia explains the high level of addiction in WoW:
Anything we spend time doing to the point of putting aside Bible study and prayer should be a signal to us that there is a problem. It is like recognizing hunger pain as a signal to eat. Time is limited. It is like a bandwith. Use it well.
And yet, it is possible to play WoW without being "addicted" to it. It also is possible to teach your children how to do things in moderation as well. What you seem to be missing is that everything we have in this world is a gift of God. This includes entertainment. That's right, being able to play World of Warcraft is a gift of God, and we should recognize it as such. It is, as I have mentioned before, adiophra. And, as such, you should follow Paul's advice to not be a slave to anything. If you are incapable of playing WoW without being addicted (to be enslaved by it), then maybe it isn't the right thing for you. Much like if you are an alcoholic and have a problem with drinking. But just because there are alcoholics in the world doesn't mean that everyone who drinks alcohol is one. Even Jesus had no problem with people drinking alcohol -- he made wine out of water at a multi-day wedding celebration -- several days into the celebration (because they were running out of wine). What do you think the people at that celebration were doing in order to have used up all of the wine? Didn't Jesus effectively condone people drinking wine by making some more good wine for the party (to the point that the steward claimed that it was the best wine, and shocked it was saved until last)? So, in the case of gaming "addiction", it really is down to the personal level, it is not expressly forbidden against in the Bible, and each person should be able to choose whether to exercise his or her Christian Freedom as to whether he or she is able to play a game like that responsibly or not.

Think on These Things (Not Evil)
Thoughts are powerful; “as he thinketh, so he is” (Prov. 23:7). Where we focus or thoughts can lead to either unrest and discouragement, or spiritual thinking and peace.
Ahh yes, the whole thought-police legalism. Better be careful to not think the wrong way, or else! (rolling eyes) I'm gong to quote the same passage you do right after this, but in a better translation. Phil 4:2-9 (ESV):

Finally, brothers, whatever is true, whatever is honorable, whatever is just, whatever is pure, whatever is lovely, whatever is commendable, if there is any excellence, if there is anything worthy of praise, think about these things.What you have learned and received and heard and seen in me—practice these things, and the God of peace will be with you.
10 I rejoiced in the Lord greatly that now at length you have revived your concern for me. You were indeed concerned for me, but you had no opportunity. 11 Not that I am speaking of being in need, for I have learned in whatever situation I am to be content. 12 I know how to be brought low, and I know how to abound. In any and every circumstance, I have learned the secret of facing plenty and hunger, abundance and need. 13 I can do all things through him who strengthens me.
14 Yet it was kind of you to share my trouble. 15 And you Philippians yourselves know that in the beginning of the gospel, when I left Macedonia, no church entered into partnership with me in giving and receiving, except you only.16 Even in Thessalonica you sent me help for my needs once and again. 17 Not that I seek the gift, but I seek the fruit that increases to your credit.[d] 18 I have received full payment, and more. I am well supplied, having received from Epaphroditus the gifts you sent, a fragrant offering, a sacrifice acceptable and pleasing to God. 19 And my God will supply every need of yours according to his riches in glory in Christ Jesus. 20 To our God and Father be glory forever and ever. Amen.
So... this passage tells the congregation to think about good things (since Paul was under house arrest at the time, and was destined to eventually be killed by the Emperor), rather than being troubled about what is going on with Paul and all of Christianity at the time. It isn't telling every Christian everywhere to only think good thoughts or else. Nope, it just is telling the Philippians that Paul is doing OK, he's doing the work God sent him out to do, and they shouldn't be troubled by it. Once again, context, context, context.

As such, I find your article to be offensive in that it twists God's Word to whatever your own moralistic and legalistic view has made it to be. It condemns people who are enjoying a form of entertainment, and makes several arbitrary lines in the sand as to what is right and wrong, all without real backing of scripture to make an objective decision. I am offended that you claim that this is how a Christian would view it, because it most definitely is not a Christian viewpoint. It is a legalistic, fundamentalist view that has no solid footing against the clear words of scripture.

Further, I personally have found the following to be true for a Christian to follow with regards to World of Warcraft -- Christians are free to play or not to play it as they see fit. Christian parents are free to decide whether their children should play it or not. Christians should be wary of addiction in any form, as it is never good to become enslaved by anything, and if a specific person has the problem of video game addiction, then it might not be best for that person to play this game. But if a person doesn't have that problem, then it isn't an issue. I would recommend that Christian parents to review what is in any video game (or movie, TV show, book, song, etc.) and see if it is something that is appropriate for their children. But I'd recommend that for any parent, even non-Christian ones. I'd recommend that any parent gets involved in whatever his or her child is doing, and definitely not consider a video game like World of Warcraft as a surrogate or substitute for love. I have seen in-game a number of parent/child combinations that have worked, where both a son/daughter and father/mother have played the game together, and used it as a way to bond and enjoy time together (especially if the parent is in the military and deployed/in training somewhere, or any other reason the parent is away from the home for a while). I wouldn't be worried about the "scantily-clad females" in the game. Your child will most likely encounter far worse just walking through any amusement park, or out on a beach. Use it as a good time to teach your child about respecting a woman (or respecting herself if your daughter).

Tuesday, October 23, 2012

All Hallow's Eve

Growing up, I was raised with the understanding that Halloween was a "pagan holiday" that was to not be celebrated at all. Our Halloween "tradition" was to put blankets over every window that could possibly leak light out of them on Halloween night, all cuddle around a single, solitary lamp in the back of the house, and maybe play a board game or two just to keep busy -- never anything that was loud of course, and just wait out Halloween night in seclusion. Of course, this wasn't always the case. When my father was a pastor in Needles, we had a "Reformation Day" costume party on October 31st, and even before that, I distinctly remember trick-or-treating as a little kid dressed in... I believe a homemade Cheshire Cat costume when I was 5 or so. But my Dad had read some information about Halloween's origins, and decided that we were not going to support a pagan, anti-Christian holiday.

Now, as an adult, married, and with a family of my own, I have tackled the Halloween question anew... and as much as I respect my father, and I appreciate how blessed I was being raised in a Christian family, I will have to question his position on Halloween. Is Halloween a pagan holiday? Well, curiously enough, there was a historic pagan holiday celebrated on October 31st called Samhain. It is a harvest festival, to celebrate the start of the "darker half of the year", another year's harvest, and a time when spirits of the dead were supposed to be able to enter our world. Ok, by all accounts, this is a pagan holiday that has nothing to do with Christianity.

However, another Holiday, originally celebrated in April but eventually migrated to November 1st came into being -- All Saints Day, which was celebrated by the Christian church only a handful of centuries after Christ's Ascension. Eventually, there was an influence of the popular purgatory heresy that affected the night before All Saint's Day, called All Saint's Eve (or All Hallow's Eve), which would be celebrated for the people supposedly in purgatory as a chance for those trapped in purgatory to be able to right whatever wrongs they did that keeps them bound in purgatory and out of heaven -- essentially the spirits of the dead being able to enter our world. Ok, so this also really isn't Christian, it is a heretical teaching that was infecting Christianity at the time (and still does in Roman Catholicism and Eastern Orthodoxy).

At some point, these two holidays began to influence each other (as there are obvious similarities, and it is likely that "All Hallow's Eve" was originally influenced by Samhain in the Celtic world and eventually spread throughout all of the Christian world at the time). As time progressed, the religious influences were washed out from the combined holiday, now called Halloween, and it became more strongly influenced by the secular world than either religious origin. About a hundred to a hundred and fifty years ago, the modern rebirth of paganism (neopaganism & Wicca) began to want to take claim over the holiday as their own -- with some justification, as they claim to be the continuation of the ancient pagan world that created the Samhain festival, yet the holiday really has nothing much to do with the old ways any more. It has become less involved with the seasons, and more involved with secular family fun (trick-or-treating, costume parties, decorations) and sudden shocks/startles from surprises/scares.

To make the claim that Halloween is a pagan holiday seems a bit of a stretch. It has pagan & heretical origins, that is true, but those origins really have no influence on the holiday as it is celebrated today. What does the Bible say about things like this? Well, there is a similar situation covered by Paul in 1 Corinthians 8 (ESV):
Food Offered to Idols
Now concerning food offered to idols: we know that “all of us possess knowledge.” This “knowledge” puffs up, but love builds up. If anyone imagines that he knows something, he does not yet know as he ought to know. But if anyone loves God, he is known by God.

Therefore, as to the eating of food offered to idols, we know that “an idol has no real existence,” and that “there is no God but one.” For although there may be so-called gods in heaven or on earth—as indeed there are many “gods” and many “lords”— yet for us there is one God, the Father, from whom are all things and for whom we exist, and one Lord, Jesus Christ, through whom are all things and through whom we exist.
However, not all possess this knowledge. But some, through former association with idols, eat food as really offered to an idol, and their conscience, being weak, is defiled. Food will not commend us to God. We are no worse off if we do not eat, and no better off if we do. But take care that this right of yours does not somehow become a stumbling block to the weak. 10 For if anyone sees you who have knowledge eating in an idol’s temple, will he not be encouraged, if his conscience is weak, to eat food offered to idols? 11 And so by your knowledge this weak person is destroyed, the brother for whom Christ died. 12 Thus, sinning against your brothers and wounding their conscience when it is weak, you sin against Christ. 13 Therefore, if food makes my brother stumble, I will never eat meat, lest I make my brother stumble.
I believe this has a practical application to Halloween. This particular passage dealt with a problem that Greek Christians were having to deal with -- meat that was being sold in the marketplace which was being offered to pagan gods. The meat was consecrated and sacrificed to these gods, which were obviously anti-Christian, and the Greek Christians were concerned that eating this food offered to other gods would be wrong. Paul points out that these other "gods" aren't real, and the food is still just food, regardless of whatever people may have done beforehand. Armed with the knowledge that these other "gods" aren't real, it becomes a non-issue, and Paul states that it is perfectly fine to eat the food from the marketplaces that were sacrificed in this manner without it being a sin or wrong.

However he also says that there are some that are "weaker" -- that is, it defiles their conscience to eat this meat -- Christians who still feel that it is wrong to eat the meat sacrificed to other "gods", and it would be wrong to force these Christians with a weakness in their faith (that is, they don't fully trust that there is only one God, that the other "gods" don't exist, or that God might get upset if they eat it) to eat the food sacrificed to false "gods", because it can make them stumble -- such a thing could shatter their weak faith and make them lose their faith all together and no longer believe in Christ and what he did for them. Making anyone lose their faith is a very bad thing.

How does this have application to Halloween? Well, the celebration of Halloween has roots in pagan and heretical beliefs and practices, but those aspects simply don't apply in the modern, common celebration of Halloween -- the holiday itself takes the place of the "meat sacrificed to idols" that Paul wrote about. The purgatory heresy and the pagan gods & beliefs are all false, and don't do anything to a Christian -- you aren't "guilty by association" by celebrating a holiday that may have once been a pagan festival, but no longer has that pagan element in the holiday. It is sterilized (as the commercial/secular world likes to do with anything religious -- purge all religious overtones completely from it, for reference, look at what has happened to Christmas) from the pagan roots, and no longer is connected to it. A Christian doesn't need to fear Halloween.

What about this verse in the bible: "whatever is true, whatever is honorable, whatever is just, whatever is pure, whatever is lovely, whatever is commendable, if there is any excellence, if there is anything worthy of praise, think about these things."? Obviously this says to stay away from Halloween, because Halloween isn't pure, lovely, commendable, etc. right? Well, this verse comes from Philippians 4:8... and it is often taken way out of context and applied to things it has no business being applied to. Let's look at Philippians 4:2-9 (ESV):

Exhortation, Encouragement, and Prayer
I entreat Euodia and I entreat Syntyche to agree in the Lord. Yes, I ask you also, true companion, help these women, who have labored side by side with me in the gospel together with Clement and the rest of my fellow workers, whose names are in the book of life.

Rejoice in the Lord always; again I will say, rejoice. Let your reasonableness be known to everyone. The Lord is at hand; do not be anxious about anything, but in everything by prayer and supplication with thanksgiving let your requests be made known to God. And the peace of God, which surpasses all understanding, will guard your hearts and your minds in Christ Jesus.
Finally, brothers, whatever is true, whatever is honorable, whatever is just, whatever is pure, whatever is lovely, whatever is commendable, if there is any excellence, if there is anything worthy of praise, think about these things. What you have learned and received and heard and seen in me—practice these things, and the God of peace will be with you.

This passage is dealing with issues directly in the congregation at Philippi. First dealing with some squabble going on between two women in that church, and then encouraging the Philippians to be peaceful and to pray about good things (to not get caught up in whatever fight was going on between the afore-mentioned ladies). Verse 8 isn't a rule for everyone to yank out of this context and then apply to everything in life no matter what. It is a nice sentiment, but really it has to do with church practice and prayer, based on the context, not a rule to measure things against to determine if Christians should or shouldn't be involved with it. It has no application to Halloween whatsoever.

Friday, April 6, 2012

Music Playlist Generator, Part 2

Part 1

Now that I had a XML file representing everything in my playlist, I needed to be able to actually use it within my Python script... so I could then, in turn, generate my daily playlist. I'm using PyPy, which is a rather unique variation of Python that can drastically speed up the performance of Python... so I figured why not? Anyway, this does pose some limitations on how I can proceed, as a number of Python XML parsing libraries are built in C with Python hooks, something PyPy doesn't handle gracefully (yet). So I decided to stick with what I had in the Python Standard Library, which included the ever so convenient ElementTree that quickly and easily allows for me to process an XML file.

Of course, I needed to make a Python object to load the tracks into, so first off, I built my Track object:

class Track:
        __slots__ = ('disc','discTotal','track','trackTotal','title','album','year','artist','albumArtist','genre','flags','filename','fileDate','length','size','channels','sampleRate','bitRate','fileType','fileTag','added','played')
        def __init__(self):
                self.disc = 0
                self.discTotal = 0
                self.track = 0
                self.trackTotal = 0
                self.title = ""
                self.album = ""
                self.year = ""
                self.artist = ""
                self.albumArtist = ""
                self.genre = ""
                self.flags = []
                self.filename = ""
                self.fileDate = ""
                self.length = 0
                self.size = 0
                self.channels = ""
                self.sampleRate = ""
                self.bitRate = ""
                self.fileType = ""
                self.fileTag = ""
                self.added = datetime.today()
                self.played = []

        def toTuple(self):
                dat = (
                        self.disc, self.discTotal, self.track, self.trackTotal,
                        self.title, self.album, self.year, self.artist,
                        self.albumArtist, self.genre, self.flags, self.filename,
                        self.fileDate, self.length, self.size, self.channels,
                        self.sampleRate, self.bitRate, self.fileType,
                        self.fileTag
                )

                return dat

        def __eq__(self, other):
                if 'toTuple' not in dir(other):
                        return False
                return self.toTuple().__eq__(other.toTuple())

        def __ne__(self, other):
                if 'toTuple' not in dir(other):
                        return False
                return self.toTuple().__ne__(other.toTuple())

        def __lt__(self, other):
                if 'toTuple' not in dir(other):
                        return False
                return self.toTuple().__lt__(other.toTuple())

        def __le__(self, other):
                if 'toTuple' not in dir(other):
                        return False
                return self.toTuple().__le__(other.toTuple())
        def __gt__(self, other):
                if 'toTuple' not in dir(other):
                        return False
                return self.toTuple().__gt__(other.toTuple())

        def __ge__(self, other):
                if 'toTuple' not in dir(other):
                        return False
                return self.toTuple().__ge__(other.toTuple())

        def checkSum(self):
                return hashlib.sha512(str(self.toTuple())).hexdigest()

        def copy(self, other):
                log.info("Changing \"%s\":", self.filename)

                logLine = "\t%s (%s -> %s)"
                logLineQ = "\t%s (\"%s\" -> \"%s\")"

                if self.disc != other.disc:
                        log.info(logLine, "Disc", str(self.disc), str(other.disc))
                        self.disc = other.disc

                if self.discTotal != other.discTotal:
                        log.info(logLine, "Disc Total", str(self.discTotal), str(other.discTotal))
                        self.discTotal = other.discTotal

                if self.track != other.track:
                        log.info(logLine, "Track", str(self.track), str(other.track))
                        self.track = other.track

                if self.trackTotal != other.trackTotal:
                        log.info(logLine, "Track Total", str(self.trackTotal), str(other.trackTotal))
                        self.trackTotal = other.trackTotal

                if self.title != other.title:
                        log.info(logLineQ, "Title", str(self.title), str(other.title))
                        self.title = other.title

                if self.album != other.album:
                        log.info(logLineQ, "Album", str(self.album), str(other.album))
                        self.album = other.album

                if self.year != other.year:
                        log.info(logLine, "Year", str(self.year), str(other.year))
                        self.year = other.year

                if self.artist != other.artist:
                        log.info(logLineQ, "Artist", str(self.artist), str(other.artist))
                        self.artist = other.artist

                if self.albumArtist != other.albumArtist:
                        log.info(logLineQ, "Album Artist", str(self.albumArtist), str(other.albumArtist))
                        self.albumArtist = other.albumArtist

                if self.genre != other.genre:
                        log.info(logLineQ, "Genre", str(self.genre), str(other.genre))
                        self.genre = other.genre

                if self.flags != other.flags:
                        log.info(logLineQ, "Flags", str(self.flags), str(other.flags))
                        self.flags = other.flags

                if self.filename != other.filename:
                        log.info(logLineQ, "Filename", str(self.filename), str(other.filename))
                        self.filename = other.filename

                if self.fileDate != other.fileDate:
                        log.info(logLine, "File Date", str(self.fileDate), str(other.fileDate))
                        self.fileDate = other.fileDate

                if self.length != other.length:
                        log.info(logLine, "Length", str(timedelta(0, self.length)), str(timedelta(0, other.length)))
                        self.length = other.length

                if self.size != other.size:
                        log.info(logLine, "Size", str(self.size), str(other.size))
                        self.size = other.size

                if self.channels != other.channels:
                        log.info(logLineQ, "Channels", str(self.channels), str(other.channels))
                        self.channels = other.channels

                if self.sampleRate != other.sampleRate:
                        log.info(logLineQ, "Sample Rate", str(self.sampleRate), str(other.sampleRate))
                        self.sampleRate = other.sampleRate

                if self.bitRate != other.bitRate:
                        log.info(logLineQ, "Bit Rate", str(self.bitRate), str(other.bitRate))
                        self.bitRate = other.bitRate

                if self.fileType != other.fileType:
                        log.info(logLineQ, "File Type", str(self.fileType), str(other.fileType))
                        self.fileType = other.fileType

                if self.fileTag != other.fileTag:
                        log.info(logLineQ, "File Tag", str(self.fileTag), str(other.fileTag))
                        self.fileTag = other.fileTag


This basically is an object that just holds onto the various XML sub-elements for the XML track into a single object for me to work with. It has a couple of other pieces, namely it some some overriding for comparison operators, has a way to copy a track from another track, and has a way to produce a tuple representation of the object that I use with the difflib for figuring out when two tracks are most likely the same (explained later).

Then, there is the actual import function:

def importData():
        """
        Imports a new set of data from the XML file.
        """
        listIter = ET.iterparse(XML_FILENAME)

        tracks = {}
        track = Track()
        trackCat = None

        for listItem in listIter:
                trackElem = listItem[1]
                if trackElem.tag == "track":
                        if trackCat not in tracks:
                                tracks[trackCat] = []
                        tracks[trackCat].append(track)
                        track = Track()
                elif trackElem.tag == "discnumber":
                        track.disc = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "totaldiscs":
                        track.discTotal = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "tracknumber":
                        track.track = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "totaltracks":
                        track.trackTotal = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "title":
                        if trackElem.text == None:
                                track.title = u''
                        else:
                                track.title = trackElem.text

                elif trackElem.tag == "album":
                        if trackElem.text == None:
                                track.album = u''
                        else:
                                track.album = trackElem.text

                elif trackElem.tag == "year":
                        if trackElem.text == None:
                                track.year = u''
                        else:
                                track.year = trackElem.text

                elif trackElem.tag == "artist":
                        if trackElem.text == None:
                                track.artist = u''
                        else:
                                track.artist = trackElem.text

                elif trackElem.tag == "albumartist":
                        if trackElem.text == None:
                                track.albumArtist = u''
                        else:
                                track.albumArtist = trackElem.text

                elif trackElem.tag == "genre":
                        if trackElem.text == None:
                                track.genre = u''
                        else:
                                track.genre = trackElem.text

                elif trackElem.tag == "category":
                        if trackElem.text == None:
                                tracvCat = u''
                        else:
                                trackCat = trackElem.text

                elif trackElem.tag == "flags":
                        if trackElem.text == None:
                                track.flags = []
                        else:
                                track.flags = trackElem.text.split(",")

                elif trackElem.tag == "filename":
                        if trackElem.text == None:
                                track.filename = u''
                        else:
                                track.filename = trackElem.text

                elif trackElem.tag == "filedate":
                        track.fileDate = datetime.strptime(trackElem.text, '%m/%d/%Y %I:%M:%S %p') if trackElem.text != None else None

                elif trackElem.tag == "length":
                        track.length = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "size":
                        track.size = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "channels":
                        if trackElem.text == None:
                                track.channels = u''
                        else:
                                track.channels = trackElem.text

                elif trackElem.tag == "samplerate":
                        track.sampleRate = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "bitrate":
                        track.bitRate = int(trackElem.text) if trackElem.text != None else None

                elif trackElem.tag == "type":
                        if trackElem.text == None:
                                track.fileType = u''
                        else:
                                track.fileType = trackElem.text

                elif trackElem.tag == "tag":
                        if trackElem.text == None:
                                track.fileTag = u''
                        else:
                                track.fileTag = trackElem.text

        for trackList in tracks.values():
                random.shuffle(trackList)

                for track in trackList:
                        track.added = datetime.today()

        return tracks
This does two things, it rapidly loads every track from the XML file, and then... because I have had to rebuild my datafile a number of times and don't want to keep playing the same set of tracks (through the LRA process explained later), I shuffle the tracks and assign the added date in random order.

Once this is completed, I am able to use the tracks in Python using the tracks object I built. I partition my music list along "category" lines, which I use the "categorygroup" or other equivalent tags within my music to identify what goes to which category. Everything about my playlist works from this set of categories -- no song exists in two categories, so it really does a complete partitioning of my music, which means I can process each category completely independent of each other... something very useful later in the code.

Wednesday, April 4, 2012

Music Playlist Generator... or how I like to make things complicated...


Well, more or less. See, I have this collection of music I've transferred from a whole lot of CD's I bought over the years onto my computer. I had an iPhone which I had used to play that music by using iTunes to create a dynamic playlist, and load that playlist onto my phone. Well... it worked, other than the fact that I can't use iTunes on Linux and such. So... I decided to go with an alternate plan -- stream the music and use a music player to listen to it. My problem was, how to make my music playlist with the same functionality that iTunes provided me.

First, I used Firefly Media Server... which became quite a hassle to work with, so I switched to Logitech's SqueezeBox Server (with an external MySQL database backend). I then built some stored procedures to generate the playlist directly into the database.... which really didn't work very well. So I built a Google AppEngine web app that would do most of the processing... which worked pretty well but was a pain to keep up. And... SqueezeBox Server died on me, and I didn't want to mess with it any more. After some searching, I settled on something simple -- MusicPlayer Daemon, with its own internal HTTP streaming output. Using this, I could once again access my music as a stream -- a continuous stream that works all the time. And, it is driven by a playlist I can control and manipulate with Python (thanks to python-mpd2). After wiring everything up to provide an encrypted channel with login credentials being required to reach the stream (thanks to squid3 acting as a reverse proxy and upgrading http to https traffic automatically), I finally had a solution. I'd just write my playlist generator as a python script.

In comes TagScanner -- a beautiful program (if a little cumbersome to learn how it works at first) which lets me clean up and organize my music & all of the music tags easily. It also provides a way for me to export my list of music to XML... which I use as the starting point for my python script. TagScanner lets me build the XML file with the following:
# Tagscanner export script

$file_name TrackList.xml
$file_notes XML file with full information
$file_encoding utf-8
$file_writebom 1
$file_ishtml 1
$file_relativepaths 1

$document_open

<tracklist>
$select %_index%,0
 <track>
  <discnumber>%disc%</discnumber>
  <totaldiscs>%totaldiscs%</totaldiscs>
  <tracknumber>%track%</tracknumber>
  <totaltracks>%totaltracks%</totaltracks>
  <title>%title%</title>
  <album>%album%</album>
  <year>%year%</year>
  <artist>%artist%</artist>
  <albumartist>%albumartist%</albumartist>
  <genre>%genre%</genre>
  <category>%contentgroup%</category>
  <flags>%comment%</flags>
  <filename>%fullfilenameext%</filename>
  <filedate>%_filedate%</filedate>
  <length>%_length_sec%</length>
  <size>%_filesize_bytes%</size>
  <channels>%_channels%</channels>
  <samplerate>%_samplerate%</samplerate>
  <bitrate>%_bitrate%</bitrate>
  <type>%_codec%</type>
  <tag>%_tagtype%</tag>
 </track>
$endselect
</tracklist>
$document_close
I'm not going to go into details as to what all of this means, but basically it generates the entire list of tracks into an XML file that I can parse. And thus my journey of building my playlist begins.

The next step was to be able to easily update my music list. Which comes down to a nice program called QtdSync, which uses the tried and true rsync algorithm for incremental updates to provide what I need. I simply sync my music folders on my portable hard drive with a directory on my server at home, and it does the nasty work of figuring out how to only update what has changed (because I don't want to have to re-upload my entire music library every time I make a change). Conveniently, this makes me have the same directory structure on my portable drive as the server, which makes the XML file above provide the same relative paths to my files.

On Part 2, I will explain the process I use for importing the XML file into something usable in my python script...

Thursday, March 29, 2012

Squid 3.1, mod_python, and how I have fixed my reverse proxy redirects...

Ahh squid, how I love to hate to love you... most of the time, you work great for me. But sometimes... grr. Anyway...

So here's my problem -- have squid (3.1.18) being used in both a transparent and reverse proxy configuration (transparent proxy for everything inside my network going out to the internet, reverse proxy to redirect certain CNAME virtual hosts to my various other servers on my network). However, I do not want to permit http traffic to go through the reverse proxy peers, I want everything https. No problem, right? Simple configuration directive for just one of my cache peers:

cache_peer 10.18.75.1 parent 80 0 no-query originserver login=PASS name=xlorep
acl sites_xlorep url_regex ^https://xlorep\.darkhelm\.org
cache_peer_access xlorep allow sites_xlorep
http_access allow sites_xlorep
acl http_xlorep url_regex ^http://xlorep\.darkhelm\.org
http_access deny http_xlorep
deny_info https://xlorep.darkhelm.org/ http_xlorep 

Basically, it uses a couple regular expressions to figure out when someone is accessing the xlorep.darkhelm.org host. If it is https, it connects to that server. If it is http, if denies access, and sends a HTTP redirect command to send the person to the https equivalent.

Perfectly fine configuration, unless the person wants to, let's say, go to http://xlorep.darkhelm.org/some/other/path.html. Rather than going to https://xlorep.darkhelm.org/some/other/path.html, the person is sent to https://xlorep.darkhelm.org/. Because squid 3.1 is too dumb to do anything better. For almost every request I use, this is *perfectly fine* -- most of my redirects go to sites that all handle themselves in a friendly way to this kind of scenario.

Enter mpd's client175, and suddenly this causes a problem. For some reason, client175 likes to send links to the unsecured version of the URL (http rather than https), which gets hosed because of the deny_info redirecting going on in my squid configuration. So... client175 becomes only partway functional. I needed to find some way to properly change the URL to the exact https equivalent of the http url.

Well, I found out that squid 3.2 did exactly that. Oh, other than it being beta software and completely crippling my configuration from the ground up preventing any http to https redirects from happening, even those from the transparent proxy side of things... so... 3.2 didn't work.

Solution #2 came to me rather simply -- I write a simple redirect script that could do the work for me, and put the URL for that in my deny_info directive above. So... after installing mod_python into my in-house apache server (why mod_python? because I like python -- same thing for client175, it is a python-based mpd client). So... get mod_python configured with a simple addition to the default site for apache:

<Directory /var/www/py>
        AddHandler mod_python .py
        PythonHandler mod_python.publisher
        PythonDebug On
</Directory>

And then write a simple python script to solve my problem:

from mod_python import apache, util

import re, urllib

def redir(req):
        req.log_error('handler')
        reqURI = req.unparsed_uri

        patt = re.compile(r"^\/py\/redirect\.py\/redir\?uri=")

        reSearch = re.search(patt, reqURI)

        oldURL = urllib.unquote(reqURI[reSearch.end():])

        newURL = re.sub(r"^http:\/\/","https:\/\/",oldURL)

#       req.content_type = 'text/html'
#       req.send_http_header()
#       req.write('')
#       req.write('' + reqURI + '<br />')
#       req.write('' + newURL + '<br />')
#       req.write('')
#       return apache.OK
        return util.redirect(req, newURL)

I configure this to be handled through redirect.darkhelm.org, and it basically translates a given URL parameter from http to https. I can then go back to that squid configuration, and change it as follows:

cache_peer 10.18.75.1 parent 80 0 no-query originserver login=PASS name=xlorep
acl sites_xlorep url_regex ^https://xlorep\.darkhelm\.org
cache_peer_access xlorep allow sites_xlorep
http_access allow sites_xlorep
acl http_xlorep url_regex ^http://xlorep\.darkhelm\.org
http_access deny http_xlorep
deny_info https://redirect.darkhelm.org/py/redirect.py/redir?uri=%s http_xlorep 

And now, if someone goes to http://xlorep.darkhelm.org/some/other/path.html they get redirected to https://xlorep.darkhelm.org/some/other/path.html -- exactly what I want. And it makes client175 work perfectly! As we used to say in the Army, HOOAH! problem solved.