Python noob help
-
I'm doing a little side-project in python (flask + pymongo). Just for fun, and as an excuse to learn the language.
As usual, I'm down with basic programming, but I'm struggling with what could be considered "best practices" in structuring the code.
Here's where I'm stuck:
In my main module, I'm doing something like this:
(app/__init.py)
app = Flask(__name__, instance_relative_config=True) app.config.from_object('default_config') app.config.from_pyfile('config.py', silent=True)
Included in this config are the settings for my mongo db. In a different module I am thinking to put my business logic / data access methods.
(app/data/users.py)
from . import db def list_all_users(): return db.users.find()
So all I need now is to initialize the shared db object in the module's
__init__.py
. A naive solution would be something like this:(app/data/__init.py)
from app import app config = app.config['MONGODB'] client = MongoClient(config['host'], config['port']) db = client['cartman_db']
This is, of course, terrible. Why would a db module know about my app module? It should just be given (you might even say injected) the info it needs.
However, Stack Overflow says dependency injection in python world isn't the darling it is in C#/Java land and seems like overkill for something this simple.
So I tried doing this:
(app/__init.py)
# other init stuff... import app.data as data data.connect(**app.config['MONGODB'])
(app/data/__init.py)
client = None db = None def connect(host='localhost', port=27017): global client, db client = MongoClient(host, port) db = client['cartman_db']
But that, of course, screws with my import strategy in
app/data/users.py
(it getsNone
values initially and doesn't update).So now, I've started looking into singletons or wrapping these globals into dictionaries, or having a "getter" function... but honestly, this all seems hacky as hell.
Am I even on the right track? I can probably make this work, but I don't think I can make myself like it.
What is the proper way to do this in python?
-
I'm no expert on python best practices, but when i have that sort of shared resource problem i do tend to reach for a singleton pattern.
have a method produce a singleton on demand for your shared resource and then just include the static method in a utils or other class and call it when you want the resource. if it's already been constructed it hands it to you, or creates one and saves it for next time too if it hasn't already been used.
it might be a bit overkill and possibly not the best pattern but it's one i know of off the top of my head and it gets the job done correctly with little additional thought on my part.
-
Yeah, that would be something like this:
(app/data/__init.py)
_client = None _db = None def connect(host='localhost', port=27017): global _client, _db _client = MongoClient(host, port) _db = _client['cartman_db'] def db(): if not _db: raise Exception('Not connected') return _db
Then, from DAL methods:
(app/data/users.py)
from . import db def list_all_users(): return db().users.find()
Or do it with a real singleton object. Still seems hacky.
-
it is a bit.
gets the job done though.
-
I am not a python, but I would probably make a class that takes a config file in its constructor, and has a map/dictionary/hash-like interface. Maybe even go as far as creating an
IConfigurable
and having your config class implement it. Then any other classes can take anIConfigurable
in their constructors, and they can query it for their configuration state.
Here, some vaguely C#-ish pseudocode—it should be easy enough to translate into snake language:
Then, in the startup file:
interface IConfigurable {
IConfigurable subConfig(string name);
string operator[](string name);
}
later in the application:
main() {
IConfigurable config = new IniConfig("app.ini");
Application myApp = new Application(config);
while(myApp.update()) {} // or whatever the main loop looks like...
//...
}
Application::Application(IConfiguration config) {
m_db = new Database(config.subConfig("database"));
m_socket = new Socket(config.subConfig("socket"));
//...
}
-
Not really KISS, since my flask app is already loading and parsing all the configuration. I would do it similar to what you're suggesting in C*, but python seems to have a different philosophy. I'm sort of trying to "go native" here as much as possible.
Ok, here's where I'm at now.
(app/data/__init__.py)
from pymongo import MongoClient class _DataContext(object): _client = None _db = None def connect(self, host='localhost', port=27017): self._client = MongoClient(host, port) self._db = self._client['cartman_db'] @property def db(self): if not self._db: raise Exception('Not connected') return self._db def __getattr__(self, item): return self.db[item] dc = _DataContext()
So, _DataContext is a private class that I'm using as a singleton. It in itself is not a Singleton, ensuring I can mess with it in the tests I will probably never write.
__getattr__
is an extra touch enabling me to use it like this:(app/__init__.py)
from app.data import dc dc.connect(**app.config['MONGODB'])
(app/data/users.py)
from . import dc def list_all_users(): return dc.users.find()
This feels "right" in terms of interface. Customizable enough and elegant to use.
Thoughts? Hidden gotchas? Performance concerns?
-
I guess my core point was: don't use Singletons, always pass a configuration object into each component's ctor. (Having said that, if your config is unlikely to change throughout the app's lifetime, you can probably get away with using a singleton even though it's morally wrong.)
You already have
dc.
as the root of all config lookups, sodc
may as well be local to the ctor, or a member variable on the class, rather than being some mysterious object that appears in scope via chicanery.Python makes you pass
self
into methods, doesn't it? So I could argue that passing in thedc
is Pythonic :<vvx>)Filed under: $0.02
-
You already have dc. as the root of all config lookups, so dc may as well be local to the ctor, or a member variable on the class, rather than being some mysterious object that appears in scope via chicanery.
I'm still learning, but I'm pretty sure that's not how python works. There are no "super-globals", all globals are limited to their module. So things never appear mysteriously, you always have to explicitly import them.
from app.data import dc
See? Also, it seems that modules are already sort of singletons in a sense that python runtime instantiates them as singletons and that's what you get from
import
calls. So you don't really need a singleton class as such. A common practice (it seems from SO) is to just create an instance inside the module's__init__
and useimport
statements as a singleton getter, to pass the instance around. Which is what I'm doing above.I think you're sort of projecting your C* "instincts" and patterns into a very different language. That said, you might also be right that I should just find a DI container instead of patching things together with native modules.
However, baring a slap from a 'native' python guru, I think I'll keep the pattern I have now.
-
Fair enough. I'm probably being "the opposite of useful".
-
Fair enough. I'm probably being "the opposite of useful".
No prob, it helps me learn when I'm explaining things. Blind leading the blind won't get far, but at least they are on a journey.
-
Why are your prefixing stuff with an underscore?
-
Python doesn't have real private / protected syntax, so that is a convention. A single underscore "
_
" in modules and "__
" in classes.There are also language constructs that will act differently depending on the underscoring. For example, if I do:
from app.data import *
...I will get
dc
variable, but not_DataContext
class. I can still access it, though, like this:import app.data as data dc = data._DataContext()
-
Cheers I am a total noob with python and been writing a few scripts (boring data import stuff), though the stuff I happened to have written are scripts.
-
Python doesn't have real private / protected syntax, so that is a convention. A single underscore "_" in modules and "__" in classes.
I believe the leading
_
triggers a rewriting of the real method name to something else (that includes a munged class name IIRC?) so giving pseudo-privacy.
-
Don't put code in init.py files. Package level values yes, logic code no, and heavy (DB) code never. There are some articles explaining why but basically, it's not the correct way since this files are only to mark a directory as a package, not for code.
Now, there's Flask-PyMongo which maps & connects your models to a MongoDB. Use it, because AFAIR it also provides caching and seamless conversion.
-
I believe the leading _ triggers a rewriting of the real method name to something else (that includes a munged class name IIRC?) so giving pseudo-privacy.
That's right.
Don't put code in init.py files. Package level values yes, logic code no, and heavy (DB) code never. There are some articles explaining why but basically, it's not the correct way since this files are only to mark a directory as a package, not for code.
Got it. I'll move the
_DataContext
class to a different file.Now, there's Flask-PyMongo which maps & connects your models to a MongoDB. Use it, because AFAIR it also provides caching and seamless conversion.
I hate it. As far as I can see, all it does is fuse your db directly to your API in the worst thick-client manner possible.
From their site:
Collection.find_one_or_404(*args, **kwargs)
Find and return a single document, or raise a 404 Not Found exception if no document matches the query spec. See find_one() for detailsUgh. Also, all their examples happily dig directly into the database from the view methods. Call me old-fashioned but I need at least one layer of separation there for the peace of mind.
Also, the docs don't say anything about models or caching. Just show you a bunch of terrible shortcuts you should never take.
-
-
-
That's what you get for using MongoDB :-p
-
-
singleton pattern
I've let¹ them in our project at $work couple of years ago, before I read² Singletons: Solving problems you didn’t know you never had since 1995³. I really regret it.
Edit: Weird clipboard behaviour in chromimum strikes again. Actually did what I said in ³.
-
added wayback machine link because source site is offline
/me begins to read the article
huh... well this i will need to think on....
yes think carefully.
-
However, Stack Overflow says dependency injection in python world isn't the darling it is in C#/Java land and seems like overkill for something this simple.
Design patterns are not language-specific. Their realization might be—interface in python is just a documented method with specific arguments, many things can be simple functions in python where you'd need a class and an interface in Java and such—but the concepts remain.
I do sometimes have feeling that especially Java community has many Architecture Astronauts who overuse the patterns even in cases where they don't really bring anything and just make the code longer though. The python approach is to follow YAGNI: don't add more words until it actually gives you some benefit.
-
added wayback machine
I opened wayback machine when I was writing it and then chromium didn't select the link as I expected and I didn't notice it pasted the wrong thing. Lately chromium (might be X, but I don't recall this problem when selecting from terminal) only sets the active selection when I select something the second time. It is clearly intended to avoid wiping it with the auto-select when clicking into entry so it can be pasted over, but it ends up behaving weird.
-
However, Stack Overflow says dependency injection in python world isn't the darling it is in C#/Java land and seems like overkill for something this simple.
It's more that dependency injection frameworks aren't a darling of the Python world -- manual DI is alive and well in Python, we just don't build entire programs around DI frameworks because it makes much less sense to do so when your programming language isn't [s]a pile of verbose spew[/s]Java.See? Also, it seems that modules are already sort of singletons in a sense that python runtime instantiates them as singletons and that's what you get from import calls. So you don't really need a singleton class as such. A common practice (it seems from SO) is to just create an instance inside the module's init and use import statements as a singleton getter, to pass the instance around. Which is what I'm doing above.
QFT -- Python modules make better singletons than most instances of the Singleton pattern. Not that Singletons are all that they're chalked up to be...I believe the leading _ triggers a rewriting of the real method name to something else (that includes a munged class name IIRC?) so giving pseudo-privacy.
It's the leading __, and only inside classes. In Python, _ is used for internal methods/functions (rarely, internal variables as well), and __ is used for 'true' privacy because of the munging scheme.
-
It's more that dependency injection frameworks aren't a darling of the Python world -- manual DI is alive and well in Python, we just don't build entire programs around DI frameworks because it makes much less sense to do so when your programming language isn't a pile of verbose spewJava.
Yeah I think I'll just keep using native modules. Python has a good system, reminds me of common.js a bit (if you avoid using "
from module import *
" crap).As for the Singleton, the problem isn't that you want only one instance of a class in your app (a legit requirement), but that you want one instance in ANY app EVER. That's pushing a concern of a specific app down into a class. A wrong approach.
-
As for the Singleton, the problem isn't that you want only one instance of a class in your app (a legit requirement), but that you want one instance in ANY app EVER. That's pushing a concern of a specific app down into a class. A wrong approach.
Exactly. Singleton by configuration is fine.
-
Lately chromium (might be X, but I don't recall this problem when selecting from terminal) only sets the active selection when I select something the second time.
Hmm....I haven't noticed that, exactly, but I've learned not to rely on active selection on chrome / chromium. Even other clipboard actions are a bit wonky. When going between VMs I find myself pasting into Kate and copying from there before it will make the jump.